Image processing device and image processing method that horizontal filtering on pixel blocks

ABSTRACT

An image processing device including a decoding section configured to decode an image from an encoded stream, a determination section configured to perform determination processes of determining whether to apply a deblocking filter to neighboring blocks neighboring across a block boundary within an image to be decoded by the decoding section, a filtering section configured to apply a deblocking filter to neighboring blocks to which the determination section has determined to apply a deblocking filter, and a control section configured to allow the determination section to perform the determination processes for a vertical block boundary and a horizontal block boundary using pixels of the neighboring blocks of a reconstruct image as reference pixels.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Ser. No. 16/391,314, filedApr. 23, 2019, which is a continuation of U.S. Ser. No. 15/902,870,filed Feb. 22, 2018 (now U.S. Pat. No. 10,362,318), which is acontinuation of U.S. Ser. No. 13/990,893, filed May 31, 2013 (now U.S.Pat. No. 9,973,763), which is a national stage of PCT/JP2011/077953,filed Dec. 2, 2011, and also claims priorities under 35 U.S.C. 119 toJapanese Application No. 2010-272907 filed Dec. 7, 2010, JapaneseApplication No. 2011-004391, filed Jan. 12, 2011, Japanese ApplicationNo. 2011-045652, filed Mar. 2, 2011, and Japanese Application No.2011-117557, filed May 26, 2011, the entire contents of each areincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to an image processing device and animage processing method.

BACKGROUND ART

H.264/AVC, one of standard specifications for image encoding scheme,applies a deblocking filter to a block boundary in units of blocks eachcontaining 4×4 pixels, for example, in order to prevent image qualitydegradation due to block distortion while an image is encoded. Thedeblocking filter requires a large amount of processing and may accountfor 50% of the entire processing amount in image decoding, for example.

The standards work for High Efficiency Video Coding (HEVC), anext-generation image encoding system, proposes application of thedeblocking filter in units of blocks each containing 8×8 pixels or moreaccording to JCTVC-A119 (see Non-Patent Literature 1). The techniqueproposed in JCTVC-A119 increases the block size as a minimum unit forapplying the deblocking filter to perform filtering processes inparallel on block boundaries in the same direction within one macroblock.

CITATION LIST Non-Patent Literature

Non-Patent Literature 1: K. Ugur (Nokia), K. R. Andersson (LM Ericsson),A. Fuldseth (Tandberg Telecom), “JCTVC-A119:Video coding technologyproposal by Tandberg, Nokia, and Ericsson”, Documents of the firstmeeting of the Joint Collaborative Team on Video Coding (JCT-VC),Dresden, Germany, 15-23 Apr. 2010.

SUMMARY OF INVENTION Technical Problem

However, the technique proposed in JCTVC-A119 does not solve dependencybetween the process on vertical block boundaries and the process onhorizontal block boundaries. It is still difficult to enable parallelprocessing on block boundaries in different directions within one macroblock (or one encoding unit) and enable parallel processing betweenmacro blocks. Accordingly, the above-described technique may notsuccessfully solve problems of a delay and a decrease in data rates dueto a large processing amount while the deblocking filter is applied.

The technology according to the disclosure aims at providing an imageprocessing device and an image processing method capable of providingparallel processing when a deblocking filter is applied.

Solution to Problem

According to an embodiment of the present disclosure, there is providedan image processing device including a decoding section configured todecode an image from an encoded stream, a determination sectionconfigured to perform determination processes of determining whether toapply a deblocking filter to neighboring blocks neighboring across ablock boundary within an image to be decoded by the decoding section, afiltering section configured to apply a deblocking filter to neighboringblocks to which the determination section has determined to apply adeblocking filter, and a control section configured to allow thedetermination section to perform the determination processes for avertical block boundary and a horizontal block boundary using pixels ofthe neighboring blocks of a reconstruct image as reference pixels.

The image processing device can be realized typically as an imagedecoding device for decoding an image.

Further, according to an embodiment of the present disclosure, there isprovided an image processing method including decoding an image from anencoded stream, performing determination processes of determiningwhether to apply a deblocking filter to neighboring blocks neighboringacross a block boundary within an image to be decoded, applying adeblocking filter to neighboring blocks to which it has been determinedin the determination processes to apply a deblocking filter, andcontrolling the determination processes in a manner that thedetermination processes for a vertical block boundary and a horizontalblock boundary are performed using pixels of the neighboring blocks of areconstruct image as reference pixels.

Further, according to an embodiment of the present disclosure, there isprovided an image processing device including a determination sectionconfigured to perform determination processes of determining whether toapply a deblocking filter to neighboring blocks neighboring across ablock boundary within an image to be locally decoded when an image to beencoded is encoded, a filtering section configured to apply a deblockingfilter to neighboring blocks to which the determination section hasdetermined to apply a deblocking filter, a control section configured toallow the determination section to perform the determination processesfor a vertical block boundary and a horizontal block boundary usingpixels of the neighboring blocks of a reconstruct image as referencepixels, and an encoding section configured to encode the image to beencoded using an image filtered by the filtering section.

The image processing device can be realized typically as an imageencoding device for encoding an image.

According to an embodiment of the present disclosure, there is providedan image processing method including performing determination processesof determining whether to apply a deblocking filter to neighboringblocks neighboring across a block boundary within an image to be locallydecoded when an image to be encoded is encoded, applying a deblockingfilter to neighboring blocks to which it has been determined in thedetermination processes to apply a deblocking filter, controlling thedetermination processes in a manner that the determination processes fora vertical block boundary and a horizontal block boundary are performedusing pixels of the neighboring blocks of a reconstruct image asreference pixels, and encoding the image to be encoded using an imagefiltered by the deblocking filter.

Advantageous Effects of Invention

As described above, the image processing device and the image processingmethod according to the present disclosure further improves parallelprocessing when a deblocking filter is applied.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing an example of a configuration of animage encoding device according to an embodiment.

FIG. 2 is a block diagram showing an example of a configuration of animage decoding device according to an embodiment.

FIG. 3 is an explanatory diagram showing an example of neighboringpixels around a boundary.

FIG. 4 is an explanatory diagram illustrating reference pixels duringfiltering need determination processes according to an existingtechnique.

FIG. 5 is an explanatory diagram illustrating pixels updated byfiltering processes.

FIG. 6 is an explanatory diagram illustrating identification of edgesfor description of the embodiment.

FIG. 7 is an explanatory diagram illustrating a parallel processaccording to an existing technique.

FIG. 8 is a first explanatory diagram illustrating dependency betweenprocesses according to an existing technique.

FIG. 9 is a second explanatory diagram illustrating dependency betweenprocesses according to an existing technique.

FIG. 10 is an explanatory diagram illustrating a sequence of processesaccording to an existing technique.

FIG. 11 is a first explanatory diagram illustrating reference pixelsduring filtering need determination processes according to a firstworking example.

FIG. 12 is a second explanatory diagram illustrating reference pixelsduring the filtering need determination processes according to the firstworking example.

FIG. 13 is an explanatory diagram illustrating a first example ofprocess sequence.

FIG. 14 is an explanatory diagram illustrating a second example ofprocess sequence.

FIG. 15 is a block diagram illustrating a detailed configuration of adeblocking filter according to the first working example.

FIG. 16 is a block diagram illustrating a detailed configuration of adetermination section.

FIG. 17 is an explanatory diagram illustrating neighboring blocks arounda slice boundary.

FIG. 18 is an explanatory diagram illustrating a first example of asequence of processes for each slice.

FIG. 19 is an explanatory diagram illustrating a second example of asequence of processes for each slice.

FIG. 20 is a flowchart illustrating a first example of a process flowfor the deblocking filter according to an embodiment.

FIG. 21 is a flowchart illustrating a second example of a process flowfor the deblocking filter according to an embodiment.

FIG. 22 is a flowchart illustrating a flow of filtering needdetermination processes according to an embodiment.

FIG. 23 is a block diagram illustrating a detailed configuration of thedeblocking filter according to a second working example.

FIG. 24 is an explanatory diagram illustrating first and second examplesof a determination technique provided by the second working example.

FIG. 25 is an explanatory diagram illustrating third and fourth examplesof the determination technique provided by the second working example.

FIG. 26 is an explanatory diagram illustrating fifth and sixth examplesof the determination technique provided by the second working example.

FIG. 27 is an explanatory diagram illustrating a process sequence foreach LCU.

FIG. 28 is a flowchart illustrating a process flow for each LCU.

FIG. 29 is an explanatory diagram illustrating an overview of a thirdworking example.

FIG. 30 is a block diagram illustrating a detailed configuration of adeblocking filter according to the third working example.

FIG. 31 is an explanatory diagram illustrating determination of a weightfor weighted average.

FIG. 32 is an explanatory diagram illustrating an example of a weightfor weighted average.

FIG. 33 is an explanatory diagram illustrating an output pixel valuefrom a calculation section according to the third working example.

FIG. 34 is an explanatory diagram illustrating a first example ofprocess sequence for comparison.

FIG. 35 is an explanatory diagram illustrating a first example ofprocess sequence provided by the third working example.

FIG. 36 is an explanatory diagram illustrating a second example ofprocess sequence for comparison.

FIG. 37 is an explanatory diagram illustrating a second example ofprocess sequence provided by the third working example.

FIG. 38 is a flowchart illustrating a first example of a process flowfor the deblocking filter according to the third working example.

FIG. 39 is a flowchart illustrating a flow of a pixel value calculationprocess shown in FIG. 38.

FIG. 40 is an explanatory diagram illustrating multiview codec.

FIG. 41 is an explanatory diagram illustrating an image encoding processaccording to an embodiment applied to multiview codec.

FIG. 42 is an explanatory diagram illustrating an image decoding processaccording to an embodiment applied to multiview codec.

FIG. 43 is an explanatory diagram illustrating scalable codec.

FIG. 44 is an explanatory diagram illustrating an image encoding processaccording to an embodiment applied to scalable codec.

FIG. 45 is an explanatory diagram illustrating an image decoding processaccording to an embodiment applied to scalable codec.

FIG. 46 is a block diagram illustrating a schematic configuration of atelevision apparatus.

FIG. 47 is a block diagram illustrating a schematic configuration of amobile phone.

FIG. 48 is a block diagram illustrating a schematic configuration of arecording/reproduction device.

FIG. 49 is a block diagram illustrating a schematic configuration of animage capturing device.

DESCRIPTION OF EMBODIMENT

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the drawings, elements that have substantiallythe same function and structure are denoted with the same referencesigns, and repeated explanation is omitted.

Description of Embodiment will be described in the following sequence.

1. Apparatus Overview

-   -   1-1. Image Encoding Device    -   1-2. Image Decoding Device

2. Existing Technique

-   -   2-1. Basic Configuration of Deblocking Filter    -   2-2. Dependency Between Processes According to an Existing        Technique

3. First Working Example

-   -   3-1. Deblocking Filter Configuration Example    -   3-2. Process Flow

4. Second Working Example

-   -   4-1. Deblocking Filter Configuration Example    -   4-2. Process Flow    -   4-3. Process Example for Each LCU

5. Third Working Example

-   -   5-1. Overview    -   5-2. Deblocking Filter Configuration Example    -   5-3. Process Sequence Example    -   5-4. Process Flow

6. Application to Various Codecs

-   -   6-1. Multiview Codec    -   6-2. Scalable Codec

7. Example Applications

8. Summing-up

1. Apparatus Overview

With reference to FIGS. 1 and 2, the following describes an overview ofan apparatus to which the technology disclosed in this specification isapplicable. The technology disclosed in this specification is applicableto an image encoding device and an image decoding device, for example.

[1-1. Image Encoding Device]

FIG. 1 is a block diagram showing an example of a configuration of animage encoding device 10 according to an embodiment. Referring to FIG.1, the image encoding device 10 includes an A/D (Analogue to Digital)conversion section 11, a reordering buffer 12, a subtraction section 13,an orthogonal transform section 14, a quantization section 15, alossless encoding section 16, an accumulation buffer 17, a rate controlsection 18, an inverse quantization section 21, an inverse orthogonaltransform section 22, an addition section 23, a deblocking filter 24 a,a frame memory 25, a selector 26, an intra prediction section 30, amotion estimation section 40, and a mode selection section 50.

The A/D conversion section 11 converts an image signal input in ananalogue format into image data in a digital format, and outputs aseries of digital image data to the reordering buffer 12.

The reordering buffer 12 reorders the images included in the series ofimage data input from the A/D conversion section 11. After reorderingthe images according to the a GOP (Group of Pictures) structureaccording to the encoding process, the reordering buffer 12 outputs theimage data which has been reordered to the subtraction section 13, theintra prediction section 30, and the motion estimation section 40.

The image data input from the reordering buffer 12 and predicted imagedata selected by the mode selection section 50 described later aresupplied to the subtraction section 13. The subtraction section 13calculates predicted error data which is a difference between the imagedata input from the reordering buffer 12 and the predicted image datainput from the mode selection section 50, and outputs the calculatedpredicted error data to the orthogonal transform section 14.

The orthogonal transform section 14 performs orthogonal transform on thepredicted error data input from the subtraction section 13. Theorthogonal transform to be performed by the orthogonal transform section14 may be discrete cosine transform (DCT) or Karhunen-Loeve transform,for example. The orthogonal transform section 14 outputs transformcoefficient data acquired by the orthogonal transform process to thequantization section 15.

The transform coefficient data input from the orthogonal transformsection 14 and a rate control signal from the rate control section 18described later are supplied to the quantization section 15. Thequantization section 15 quantizes the transform coefficient data, andoutputs the transform coefficient data which has been quantized(hereinafter, referred to as quantized data) to the lossless encodingsection 16 and the inverse quantization section 21. Also, thequantization section 15 switches a quantization parameter (aquantization scale) based on the rate control signal from the ratecontrol section 18 to thereby change the bit rate of the quantized datato be input to the lossless encoding section 16.

The quantized data input from the quantization section 15 andinformation described later about intra prediction or inter predictiongenerated by the intra prediction section 30 or the motion estimationsection 40 and selected by the mode selection section 50 are supplied tothe lossless encoding section 16. The information about intra predictionmay include prediction mode information indicating an optimal intraprediction mode for each block, for example. Also, the information aboutinter prediction may include prediction mode information for predictionof a motion vector for each block, difference motion vector information,reference image information, and the like, for example.

The lossless encoding section 16 generates an encoded stream byperforming a lossless encoding process on the quantized data. Thelossless encoding by the lossless encoding section 16 may bevariable-length coding or arithmetic coding, for example. Furthermore,the lossless encoding section 16 multiplexes the information about intraprediction or the information about inter prediction mentioned above tothe header of the encoded stream (for example, a block header, a sliceheader or the like). Then, the lossless encoding section 16 outputs thegenerated encoded stream to the accumulation buffer 17.

The accumulation buffer 17 temporarily stores the encoded stream inputfrom the lossless encoding section 16 using a storage medium, such as asemiconductor memory. Then, the accumulation buffer 17 outputs theaccumulated encoded stream at a rate according to the band of atransmission line (or an output line from the image encoding device 10).

The rate control section 18 monitors the free space of the accumulationbuffer 17. Then, the rate control section 18 generates a rate controlsignal according to the free space on the accumulation buffer 17, andoutputs the generated rate control signal to the quantization section15. For example, when there is not much free space on the accumulationbuffer 17, the rate control section 18 generates a rate control signalfor lowering the bit rate of the quantized data. Also, for example, whenthe free space on the accumulation buffer 17 is sufficiently large, therate control section 18 generates a rate control signal for increasingthe bit rate of the quantized data.

The inverse quantization section 21 performs an inverse quantizationprocess on the quantized data input from the quantization section 15.Then, the inverse quantization section 21 outputs transform coefficientdata acquired by the inverse quantization process to the inverseorthogonal transform section 22.

The inverse orthogonal transform section 22 performs an inverseorthogonal transform process on the transform coefficient data inputfrom the inverse quantization section 21 to thereby restore thepredicted error data. Then, the inverse orthogonal transform section 22outputs the restored predicted error data to the addition section 23.

The addition section 23 adds the restored predicted error data inputfrom the inverse orthogonal transform section 22 and the predicted imagedata input from the mode selection section 50 to thereby generatedecoded image data. Then, the addition section 23 outputs the generateddecoded image data to the deblocking filter 24 a and the frame memory25.

A deblocking filter 24 a performs filtering processes to decrease blockdistortion that occurs during image encoding. For example, thedeblocking filter 24 a determines necessity of filtering for each blockboundary of decoded image data supplied from an addition section 23 andapplies the deblocking filter to a boundary that is determined torequire the filter. The deblocking filter 24 a is also supplied withinformation used for the determination of filtering necessity (e.g.,mode information, transform coefficient information, and motion vectorinformation) as well as decoded image data from the addition section 23.After the filtering, the block distortion is eliminated from the decodedimage data and the deblocking filter 24 a outputs the decoded image datato frame memory 25. The process for the deblocking filter 24 a will bedescribed in detail later.

The frame memory 25 stores, using a storage medium, the decoded imagedata input from the addition section 23 and the decoded image data afterfiltering input from the deblocking filter 24 a.

The selector 26 reads, from the frame memory 25, the decoded image databefore filtering that is to be used for the intra prediction, andsupplies the decoded image data which has been read to the intraprediction section 30 as reference image data. Also, the selector 26reads, from the frame memory 25, the decoded image data after filteringto be used for the inter prediction, and supplies the decoded image datawhich has been read to the motion estimation section 40 as referenceimage data.

The intra prediction section 30 performs an intra prediction process ineach intra prediction mode, based on the image data to be encoded thatis input from the reordering buffer 12 and the decoded image datasupplied via the selector 26. For example, the intra prediction section30 evaluates the prediction result of each intra prediction mode using apredetermined cost function. Then, the intra prediction section 30selects an intra prediction mode by which the cost function value is thesmallest, that is, an intra prediction mode by which the compressionratio is the highest, as the optimal intra prediction mode. Furthermore,the intra prediction section 30 outputs, to the mode selection section50, prediction mode information indicating the optimal intra predictionmode, the predicted image data, and the information about intraprediction such as the cost function value.

A motion estimation section 40 performs an inter prediction process(prediction process between frames) based on image data for encodingsupplied from a reordering buffer 12 and decoded image data supplied viaa selector 26. For example, the motion estimation section 40 evaluatesthe prediction result of each prediction mode using a predetermined costfunction. Then, the motion estimation section 40 selects an optimalprediction mode, namely, a prediction mode that minimizes the costfunction value or maximizes the compression ratio. The motion estimationsection 40 generates predicted image data according to the optimalprediction mode. The motion estimation section 40 outputs informationabout the inter prediction such as prediction mode informationindicating the optimal intra prediction mode, the predicted image data,and the cost function value to a mode selection section 50.

The mode selection section 50 compares the cost function value relatedto the intra prediction input from the intra prediction section 30 andthe cost function value related to the inter prediction input from themotion estimation section 40. Then, the mode selection section 50selects a prediction method with a smaller cost function value, from theintra prediction and the inter prediction. In the case of selecting theintra prediction, the mode selection section 50 outputs the informationabout intra prediction to the lossless encoding section 16, and also,outputs the predicted image data to the subtraction section 13 and theaddition section 23. Also, in the case of selecting the interprediction, the mode selection section 50 outputs the information aboutinter prediction described above to the lossless encoding section 16,and also, outputs the predicted image data to the subtraction section 13and the addition section 23.

[1-2. Image Decoding Device]

FIG. 2 is a block diagram showing an example of a configuration of animage decoding device 60 according to an embodiment. With reference toFIG. 2, the image decoding device 60 includes an accumulation buffer 61,a lossless decoding section 62, an inverse quantization section 63, aninverse orthogonal transform section 64, an addition section 65, adeblocking filter 24 b, a reordering buffer 67, a D/A (Digital toAnalogue) conversion section 68, a frame memory 69, selectors 70 and 71,an intra prediction section 80, and a motion compensation section 90.

The accumulation buffer 61 temporarily stores an encoded stream inputvia a transmission line using a storage medium.

The lossless decoding section 62 decodes an encoded stream input fromthe accumulation buffer 61 according to the encoding method used at thetime of encoding. Also, the lossless decoding section 62 decodesinformation multiplexed to the header region of the encoded stream.Information that is multiplexed to the header region of the encodedstream may include information about intra prediction and informationabout inter prediction in the block header, for example. The losslessdecoding section 62 outputs the information about intra prediction tothe intra prediction section 80. Also, the lossless decoding section 62outputs the information about inter prediction to the motioncompensation section 90.

The inverse quantization section 63 inversely quantizes quantized datawhich has been decoded by the lossless decoding section 62. The inverseorthogonal transform section 64 generates predicted error data byperforming inverse orthogonal transformation on transform coefficientdata input from the inverse quantization section 63 according to theorthogonal transformation method used at the time of encoding. Then, theinverse orthogonal transform section 64 outputs the generated predictederror data to the addition section 65.

The addition section 65 adds the predicted error data input from theinverse orthogonal transform section 64 and predicted image data inputfrom the selector 71 to thereby generate decoded image data. Then, theaddition section 65 outputs the generated decoded image data to thedeblocking filter 24 b and the frame memory 69.

The deblocking filter 24 b performs filtering processes to decreaseblock distortion appearing on a decoded image. The deblocking filter 24b determines the necessity of filtering at each block boundary fordecoded image data input from the addition section 65, for example, andapplies the deblocking filter to a boundary that is determined torequire the filter. The deblocking filter 24 b is also supplied withinformation used for the determination of filtering necessity as well asdecoded image data from the addition section 65. After the filtering,the block distortion is eliminated from the decoded image data and thedeblocking filter 24 b outputs the decoded image data to the reorderingbuffer 67 and the frame memory 69. The process for the deblocking filter24 b will be described in detail later.

The reordering buffer 67 generates a series of image data in a timesequence by reordering images input from the deblocking filter 24 b.Then, the reordering buffer 67 outputs the generated image data to theD/A conversion section 68.

The D/A conversion section 68 converts the image data in a digitalformat input from the reordering buffer 67 into an image signal in ananalogue format. Then, the D/A conversion section 68 causes an image tobe displayed by outputting the analogue image signal to a display (notshown) connected to the image decoding device 60, for example.

The frame memory 69 uses a storage medium to store the decoded imagedata input from the addition section 65 before filtering and the decodedimage data input from the deblocking filter 24 b after filtering.

The selector 70 switches the output destination of the image data fromthe frame memory 69 between the intra prediction section 80 and themotion compensation section 90 for each block in the image according tomode information acquired by the lossless decoding section 62. Forexample, in the case the intra prediction mode is specified, theselector 70 outputs the decoded image data before filtering that issupplied from the frame memory 69 to the intra prediction section 80 asreference image data. Also, in the case the inter prediction mode isspecified, the selector 70 outputs the decoded image data afterfiltering that is supplied from the frame memory 69 to the motioncompensation section 90 as the reference image data.

The selector 71 switches the output source of predicted image data to besupplied to the addition section 65 between the intra prediction section80 and the motion compensation section 90 for each block in the imageaccording to the mode information acquired by the lossless decodingsection 62. For example, in the case the intra prediction mode isspecified, the selector 71 supplies to the addition section 65 thepredicted image data output from the intra prediction section 80. In thecase the inter prediction mode is specified, the selector 71 supplies tothe addition section 65 the predicted image data output from the motioncompensation section 90.

The intra prediction section 80 performs in-screen prediction of a pixelvalue based on the information about intra prediction input from thelossless decoding section 62 and the reference image data from the framememory 69, and generates predicted image data. Then, the intraprediction section 80 outputs the generated predicted image data to theselector 71.

The motion compensation section 90 performs a motion compensationprocess based on the information about inter prediction input from thelossless decoding section 62 and the reference image data from the framememory 69, and generates predicted image data. Then, the motioncompensation section 90 outputs the generated predicted image data tothe selector 71.

2. Existing Technique

[2-1. Basic Configuration of Deblocking Filter]

Generally, processes using the deblocking filter in an existing imageencoding system such as H.264/AVC or HEVC include two types ofprocesses, namely, filtering need determination processes and filteringprocesses. The following describes these two processes in HEVC, forexample.

(1) Filtering Need Determination Processes

The filtering need determination processes determine whether thedeblocking filter needs to be applied to each boundary of blocks withinan input image. Block boundaries include a vertical boundary betweenblocks horizontally adjacent to each other and a horizontal boundarybetween blocks vertically adjacent to each other. JCTVC-A119 uses ablock size of 8×8 pixels as a minimum processing unit. For example, amacro block of 16×16 pixels includes four blocks of 8×8 pixels. Theprocess is applied to one (left) vertical boundary and one (top)horizontal boundary for each block, namely, four boundaries plus fourboundaries equal to eight boundaries in total. The specification assumesthat the macro block as a technical term includes an coding unit (CU) inthe context of HEVC.

FIG. 3 is an explanatory diagram showing an example of pixels in twoblocks (neighboring blocks) Ba and Bb adjacent to each other around aboundary. The following describes the vertical boundary as an exampleand the description is obviously applicable to the horizontal boundary.The example in FIG. 3 uses symbol p_(ij) to represent a pixel in blockBa. In this symbol, i denotes a column index and j denotes a row index.The column index i is numbered as 0, 1, 2, and 3 in order (from right toleft) from the column nearest to the vertical boundary. The row index jis numbered as 0, 1, 2, . . . , 7 from the top to the bottom. The lefthalf of block Ba is omitted from the drawing. Symbol q_(kj) is used torepresent a pixel in block Bb. In this symbol, k denotes a column indexand j denotes a row index. The column index k is numbered as 0, 1, 2,and 3 in order (from left to right) from the column nearest to thevertical boundary. The right half of block Bb is omitted from thedrawing.

The following conditions can be used to determine the necessity ofapplying the deblocking filter to the vertical boundary between blocksBa and Bb shown in FIG. 3.

Determination condition of luma component (Luma) . . . . The deblockingfilter is applied if conditions A and B are both true.

Condition A:

(A1) Block Ba or Bb enters the intra prediction mode;

(A2) Block Ba or Bb has a nonzero orthogonal transform coefficient; or

(A3) |MVAx−MVBx|≥4 or |MVAy−MVBy|≥4

Condition B:

|p₂₂−2p₁₂+p₀₂|+|q₂₂−2q₁₂+q₀₂|+|p₂₅−2p₁₅+p₀₅|+|q₂₅−2q₁₅+q₀₅|<β

Condition A3 assumes a motion vector for block Ba to be (MVAx,MVAy) anda motion vector for block Bb to be (MVBx,MVBy) according to the Qpel (¼pixel) accuracy. Condition B uses β as an edge determination thresholdvalue. An initial value of β is given according to a quantizationparameter. The value for β is user-specifiable using a parameter withinthe slice header.

Determination condition of chroma component (Chroma) . . . . Thedeblocking filter is applied if condition A1 is true.

Condition A1: Block Ba or Bb enters the intra prediction mode.

As indicated by broken-line frames L3 and L6 in FIG. 4, the filteringneed determination processes on general vertical boundaries(particularly under determination condition B of luma component)reference pixels on the third and sixth rows (assuming the top row to bethe first) in each block. Similarly, the filtering need determinationprocesses on horizontal boundaries reference pixels (not shown in FIG.4) on the third and sixth columns in each block. The above-describeddetermination conditions are used to determine that the deblockingfilter needs to be applied to a boundary on which the filteringprocesses described below are performed.

(2) Filtering Processes

If it is determined that the deblocking filter needs to be applied to aboundary, the filtering processes are performed on pixels to the rightand the left of the vertical boundary and on pixels above and below thehorizontal boundary. On luma components, the filter strength is switchedbetween a strong filter and a weak filter according to pixel values.

Filtering Luma Components

Selecting the strength . . . . The filter strength is selected for eachrow or column. The strong filter is selected if all of the followingconditions C1 through C3 are satisfied. The weak filter is selected ifeven any one of the conditions is not satisfied.

(C1) d<(β>>2)

(C2) (|p_(3j)−p_(0j)|+|q_(0j)+q_(3j)|)<(β>>3)

(C3) |p_(0j)−q_(0j)|<((5t_(C)+1)>>1)

where j denotes a row index for the vertical boundary or a column indexfor the horizontal boundary.d=|p₂₂−2p₁₂+p₀₂|+|q₂₂−2q₁₂+q₀₂|+|p₂₅−2p₁₅+p₀₅|+|q₂₅−2q₁₅+q₀₅|

Weak Filtering

Δ=Clip(−t_(C),t_(C),(13(q_(0j)−p_(0j))+4(q_(1j)−p_(1j))−5(q_(2j)−p_(2j))+16)>>5))

p_(0j)=Clip₀₋₂₅₅(p_(0j)+Δ)

q_(0j)=Clip₀₋₂₅₅(p_(0j)−Δ)

p_(1j)=Clip₀₋₂₅₅(p_(1j)+Δ/2)

q_(1j)=Clip₀₋₂₅₅(q_(1j)−Δ/2)

Strong Filtering

p_(0j)=Clip₀₋₂₅₅((p_(2j)+2p_(1j)+2p_(0j)+2q_(0j)+q_(1j)+4)>>3)

q_(0j)=Clip₀₋₂₅₅((p_(1j)+2p_(0j)+2q_(0j)+2q_(1j)+q_(2j)+4)>>3)

p_(1j)=Clip₀₋₂₅₅((p_(2j)+p_(1j)+p_(0j)+q_(0j)+2)>>2)

q_(1j)=Clip₀₋₂₅₅((p_(0j)+q_(0j)+q_(1j)+q_(2j)+2)>>2)

p_(2j)=Clip₀₋₂₅₅((2p_(3j)+3p_(2j)+p_(1j)+p_(0j)+q_(0j)+4)>>3)

q_(2j)=Clip₀₋₂₅₅((p_(0j)+q_(0j)+q_(1j)+3q_(2j)+2q_(3j)+4)>>3)

where Clip(a,b,c) denotes a process to clip value c within the range ofa≤c≤b and Clip₀₋₂₅₅(c) denotes a process to clip value c within therange of 0≤c≤255.

Filtering Chroma Components

Δ=Clip(−t_(C),t_(C),((((q_(0j)−p_(0j))<<2)+p_(1j)q_(1j)+4)>>3))

p_(0j)=Clip₀₋₂₅₅(p_(0j)+Δ)

q_(0j)=Clip₀₋₂₅₅(q_(0j)−Δ)

As indicated by broken-line frames C6 through C8 and C1 through C3 inFIG. 5, the filtering processes (particularly strong filtering on lumacomponents) on general vertical boundaries update pixel values on thefirst through third and sixth through eighth columns in each block.Similarly, the filtering processes on horizontal boundaries update pixelvalues on the first through third and sixth through eighth rows in eachblock.

[2-2. Dependency Between Processes According to an Existing Technique]

For the purpose of description, as shown in FIG. 6, macro block MBx(MB0, MB1 . . . ) each having the size of 16×16 pixels includes the topleft vertical boundary represented as Vx,0, the top center verticalboundary represented as Vx,1, the bottom left vertical boundaryrepresented as Vx,2, the bottom center vertical boundary represented asVx,3, the top left horizontal boundary represented as Hx,0, the topright horizontal boundary represented as Hx,1, the left centerhorizontal boundary represented as Hx,2, and the right center horizontalboundary represented as Hx,3. Concerning boundary Z, for example, thefiltering need determination process is represented as Jz and thefiltering process is represented as Fz.

The above-described existing technique causes no dependency betweenprocesses on boundaries in the same direction within one macro block.Therefore, the technique can perform parallel filtering on verticalboundaries and horizontal boundaries within one macro block, forexample. As an example, FIG. 7 makes it clear that there is nodependency among four filtering processes F_(V0,0), F_(V0,1), F_(V0,2),and F_(V0,3) (no pixel updated redundantly) within macro block MB0 andthe filtering processes can be performed in parallel.

However, the above-described existing technique leaves the dependencybetween the filtering processes on vertical boundaries and the filteringneed determination processes on horizontal boundaries. The existingtechnique also leaves the dependency between the filtering processes onhorizontal boundaries and the filtering need determination processes onvertical boundaries. If a vertical boundary is processed prior to ahorizontal boundary, for example, the filtering need determinationprocesses need to be performed on horizontal boundaries within a givenmacro block after termination of the filtering processes on verticalboundaries. As an example, FIG. 8 shows that, within macro block MB0,filtering need determination process J_(H0,0) depends on results offiltering processes F_(V0,0) and F_(V0,1) and filtering needdetermination process J_(H0,1) depends on a result of filteringprocesses F_(V0,1). Similarly, the filtering need determinationprocesses need to be performed on vertical boundaries within a givenmacro block after termination of the filtering process on the horizontalboundary for the adjacent macro block. As an example, FIG. 9 shows thatfiltering need determination process J_(V1,0) for macro block MB1depends on results of filtering processes F_(H0,1) and F_(H0,3) formacro block MB0 and filtering need determination process J_(V1,2) formacro block MB1 depends on a result of filtering process F_(H0,3) formacro block MB0.

The existing technique involves the dependency between processes andtherefore provides parallel processing of the deblocking filter to avery limited extent even if the technique proposed in JCTVC-A119 isused.

FIG. 10 is an explanatory diagram illustrating a sequence of deblockingfilter processes according to an existing technique. The example assumesthat the deblocking filter is supplied with an image having the size of32×32 pixels. The input image includes four macro blocks MB0 through MB3each having the size of 16×16 pixels.

In FIG. 10, each broken-line frame represents a process to be performedin parallel. For example, the first step performs, in parallel,filtering need determination processes J_(V0,0), J_(V0,1), J_(V0,2) andJ_(V0,3) on four vertical boundaries in macro block MB0. The second stepperforms, in parallel, filtering processes F_(V0,0), F_(V0,1), F_(V0,2)and F_(V0,3) on four vertical boundaries in macro block MB0. Aftertermination of the second step, the third step performs, in parallel,filtering need determination processes J_(H0,0), J_(H0,1), J_(H0,2) andJ_(H0,3) on four horizontal boundaries in macro block MB0. The fourthstep performs, in parallel, filtering processes F_(H0,0), F_(H0,1),F_(H0,2) and F_(H0,3) on four horizontal boundaries in macro block MB0.After termination of the fourth step, processes (fifth to eighth steps)for macro block MB1 are performed successively. After termination of theprocesses on macro block MB1, processes (ninth to twelfth steps) formacro block MB2 are performed successively. After termination of theprocesses on macro block MB2, processes (thirteenth to sixteenth steps)for macro block MB3 are performed successively.

Such parallel processing within the limited extent cannot satisfactorilysolve the problem of delay or data rate degradation due to a largeprocessing amount when the deblocking filter is applied. Three workingexamples described below further improve parallel processing when thedefinition is applied.

3. First Working Example

[3-1. Deblocking Filter Configuration Example]

The following describes example configurations of the deblocking filter24 a for the image encoding device 10 shown in FIG. 1 and the deblockingfilter 24 b for the image decoding device 60 shown in FIG. 2 accordingto the first working example. The configurations of the deblockingfilter 24 a and the deblocking filter 24 b may be common to each other.In the following description, the deblocking filter 24 a and thedeblocking filter 24 b are generically referred to as a deblockingfilter 24 when there is no need for distinction between them.

(1) Dependency Between New Processes

According to the working example, processes using the deblocking filter24 also include two types of processes, namely, filtering needdetermination processes and filtering processes. However, the deblockingfilter 24 uses values of a reference pixel different from the existingtechnique to determine whether to apply the deblocking filter tovertical boundaries and horizontal boundaries. Specifically, for thedetermination on vertical boundaries, the deblocking filter 24 uses areference pixel, i.e., a pixel that is included in pixels of adjacentblocks around the vertical boundary and belongs to a row to which thedeblocking filter for horizontal boundaries is not applied. For thedetermination on horizontal boundaries, the deblocking filter 24 usesanother reference pixel, i.e., a pixel that is included in pixels ofadjacent blocks around the horizontal boundary and belongs to a row towhich the deblocking filter for vertical boundaries is not applied. Alsoin the following description, the deblocking filter 24 performsprocesses based on a block size of 8×8 pixels as a processing unit, forexample.

FIG. 11 is an explanatory diagram illustrating reference pixels duringthe filtering need determination processes performed by the deblockingfilter 24 on vertical boundaries. With reference to FIG. 11, macro blockMB0 has the size of 16×16 pixels. The deblocking filter 24 determineswhether to apply the filtering to four vertical boundaries of macroblock MB0 using the reference pixel that belongs to at least one of thefourth and fifth rows (L4 and L5) of each block. The deblocking filterfor horizontal boundaries is not applied to these two rows (see FIG. 9).This configuration solves the dependency between the filtering processeson horizontal boundaries and the filtering need determination processeson vertical boundaries.

FIG. 12 is an explanatory diagram illustrating reference pixels duringthe filtering need determination processes performed by the deblockingfilter 24 on horizontal boundaries. FIG. 12 also shows macro block MB0.The deblocking filter 24 determines whether to apply the filtering tofour horizontal boundaries of macro block MB0 using the reference pixelthat belongs to at least one of the fourth and fifth columns (C4 and C5)of each block. The deblocking filter for vertical boundaries is notapplied to these two columns (see FIG. 7 or 8). This configurationsolves the dependency between the filtering processes on verticalboundaries and the filtering need determination processes on horizontalboundaries.

Solving the dependency between processes can consequently parallelizefiltering need determination processes for vertical boundaries andhorizontal boundaries within one macro block. Processes can beparallelized between macro blocks. Filtering need determinationprocesses can be performed in parallel on vertical boundaries andhorizontal boundaries of all macro blocks within an input image.

FIG. 13 is an explanatory diagram illustrating a first example ofprocess sequence available on the deblocking filter 24. The example alsoassumes that the deblocking filter is supplied with an image having thesize of 32×32 pixels. The input image includes four macro blocks MB0through MB3 each having the size of 16×16 pixels.

In FIG. 13, each broken-line frame represents a process to be performedin parallel. While the example in FIG. 10 requires 16 process steps fora sequence of processes, the example in FIG. 13 aggregates the samenumber of processes into three process steps. The first step performs,in parallel, filtering need determination processes J_(V0,0) throughJ_(V3,3) and J_(H0,0) through J_(H3,3) on all vertical boundaries andall horizontal boundaries of all macro blocks MB0 through MB3. Thesecond step performs, in parallel, filtering processes F_(V0,0) throughF_(V3,3) on 16 vertical boundaries of all macro blocks MB0 through MB3.The third step performs, in parallel, filtering processes F_(H0,0)through F_(H3,3) on 16 horizontal boundaries of all macro blocks MB0through MB3. The second step and the third step may be performed in thereverse order.

The example in FIG. 13 maximizes parallelism (the quantity of processesperformed in parallel) based on parallel processing between macroblocks.

According to the example in FIG. 14, however, the deblocking filter 24can perform a process on each macro block.

The example in FIG. 14 aggregates the same number of processesillustrated in FIGS. 10 and 13 into 12 process steps. The first stepperforms, in parallel, filtering need determination processes J_(V0,0)through J_(V0,3) and J_(H0,0) through J_(H0,3) on four verticalboundaries and four horizontal boundaries of macro block MB0. The secondstep performs, in parallel, filtering processes F_(V0,0) throughF_(V0,3) on four vertical boundaries in macro block MB0. The third stepperforms, in parallel, filtering need determination processes J_(V1,0)through J_(V1,3) and J_(H1,0) through J_(H1,3) on four verticalboundaries and four horizontal boundaries of macro block MB1. The fourthstep performs, in parallel, filtering processes F_(V1,0) throughF_(V1,3) on four vertical boundaries in macro block MB1. The fifth stepperforms, in parallel, filtering processes F_(H0,0) through F_(H0,3) onfour horizontal boundaries in macro block MB0. The sixth step performs,in parallel, filtering need determination processes J_(V2,0) throughJ_(V2,3) and J_(H2,0) through J_(H2,3) on four vertical boundaries andfour horizontal boundaries of macro block MB2. The seventh stepperforms, in parallel, filtering processes F_(V2,0) through F_(V2,3) onfour vertical boundaries in macro block MB2. The eighth step performs,in parallel, filtering processes F_(H1,0) through F_(H1,3) on fourhorizontal boundaries in macro block MB1. The ninth step performs, inparallel, filtering need determination processes J_(V3,0) throughJ_(V3,3) and J_(H3,0) through J_(H3,3) on four vertical boundaries andfour horizontal boundaries of macro block MB3. The tenth step performs,in parallel, filtering processes F_(V3,0) through F_(V3,3) on fourvertical boundaries in macro block MB3. The eleventh step performs, inparallel, filtering processes F_(H2,0) through F_(H2,3) on fourhorizontal boundaries in macro block MB2. The twelfth step performs, inparallel, filtering processes F_(H3,0) through F_(H3,3) on fourhorizontal boundaries in macro block MB3. In this case, the deblockingfilter 24 can perform a process on the entire input image using processsteps fewer than those of the existing technique while the parallelismis lower than the example in FIG. 13

(2) Basic Configuration of Deblocking Filter

FIG. 15 is a block diagram illustrating a detailed configuration of thedeblocking filter 24 according to the first working example forperforming the above-described parallel processes. With reference toFIG. 15, the deblocking filter 24 includes a determination block 110, ahorizontal filtering block 130, a vertical filtering block 140, and aparallelization control section 150.

(2-1) Determination Block

The determination block 110 includes vertical boundary determinationsections 112-1 through 112-n and horizontal boundary determinationsections 114-1 through 114-n. The vertical boundary determinationsections 112 and the horizontal boundary determination sections 114 aresupplied with images input to the deblocking filter 24 and determinationinformation used to determine the necessity of filtering.

The vertical boundary determination sections 112 determine whether toapply the deblocking filter to vertical boundaries using pixel values ofreference pixels belonging to a row to which the deblocking filter forhorizontal boundaries is not applied as illustrated in FIG. 11. In thisexample, a pixel value of the reference pixel is input to the deblockingfilter 24. The vertical boundary determination sections 112 output, tothe horizontal filtering block 130, information indicating adetermination result about each vertical boundary (for example, binaryinformation of which value “1” indicates a determination result that thedeblocking filter needs to be applied).

The horizontal boundary determination sections 114 determine whether toapply the deblocking filter to horizontal boundaries using pixel valuesof reference pixels belonging to a row to which the deblocking filterfor vertical boundaries is not applied as illustrated in FIG. 12. Inthis example, a pixel value of the reference pixel is also input to thedeblocking filter 24. The determination process performed by each of thehorizontal boundary determination sections 114 is performed in parallelto the determination process performed by each of the vertical boundarydetermination sections 112. The horizontal boundary determinationsections 114 output, to the vertical filtering block 140, informationindicating a determination result about each horizontal boundary.

FIG. 16 is a block diagram illustrating a detailed configuration of eachof the vertical boundary determination sections 112 and the horizontalboundary determination sections 114. With reference to FIG. 16, eachdetermination section includes a tap constitution section 121, acalculation section 122, a threshold comparison section 123, adistortion evaluation section 124, and a filtering determination section125.

The tap constitution section 121 acquires a reference pixel value frompixel values of two neighboring blocks around a focused boundary in theinput image and constitutes a tap (a set of reference pixel values) fordetermining determination condition B for the above-described lumacomponent. For example, a vertical boundary may be focused in the blockseach of which has the size of 8×8 pixels. In this case, the tapconstitution section 121 constitutes a tap from a pixel value belongingto the fourth and/or fifth rows of two blocks at the right and left. Ifa horizontal boundary is focused, the tap constitution section 121constitutes a tap from a pixel value belonging to the fourth and/orfifth columns of two blocks at the top and bottom. The calculationsection 122 assigns the tap constituted by the tap constitution section121 to the left-hand side of the determination expression indetermination condition B and calculates an edge value to be comparedwith edge determination threshold value β. The threshold comparisonsection 123 compares the value calculated by the calculation section 122with edge determination threshold value β and outputs a comparisonresult to the filtering determination section 125.

The distortion evaluation section 124 evaluates determination conditionA of the above-described luma component using mode information (MBmode), transform coefficient information, and motion vector informationsupplied as the determination information. The distortion evaluationsection 124 outputs an evaluation result to the filtering determinationsection 125. The distortion evaluation section 124 evaluates onlydetermination condition A1 of a chroma component based on the modeinformation.

The filtering determination section 125 determines whether to apply thedeblocking filter to a focused boundary based on a comparison result ofdetermination condition B supplied from the threshold comparison section123 and an evaluation result of determination condition A supplied fromthe distortion evaluation section 124. The filtering determinationsection 125 outputs information indicating the determination result.

(2-2) Horizontal Filtering Block

Returning back to FIG. 15, the configuration of the deblocking filter 24will be described further. The horizontal filtering block 130 includeshorizontal filtering sections 132-1 through 132-n. The horizontalfiltering sections 132 are supplied with an input image and adetermination result concerning each vertical boundary from thedetermination block 110.

The horizontal filtering sections 132 apply the deblocking filter forvertical boundaries to right and left pixels around the correspondingvertical boundary if the determination result from the vertical boundarydetermination section 112 indicates that the filter needs to be applied.The horizontal filtering sections 132 output, to the vertical filteringblock 140, a pixel value after the filtering in terms of the filteredpixel or a pixel value of the input image in terms of the other pixels.

(2-3) Vertical Filtering Block

The vertical filtering block 140 includes vertical filtering sections142-1 through 142-n. The vertical filtering sections 142 are suppliedwith an input image and a determination result concerning eachhorizontal boundary from the determination block 110.

The vertical filtering sections 142 apply the deblocking filter forhorizontal boundaries to top and bottom pixels around the correspondinghorizontal boundary if the determination result from the horizontalboundary determination section 114 indicates that the filter needs to beapplied. The vertical filtering sections 142 output a pixel value afterthe filtering in terms of the filtered pixel or a pixel value suppliedfrom the horizontal filtering block 130 in terms of the other pixels. Anoutput from each of the vertical filtering sections 142 may be containedin an output image from the deblocking filter 24.

(2-4) Parallelization Control Section

The parallelization control section 150 controls the parallelism offiltering need determination processes in the determination block 110and the parallelism of filtering processes in the horizontal filteringblock 130 and the vertical filtering block 140.

For example, the parallelization control section 150 may control theparallelism of processes for each block based on an input image size.More specifically, the parallelization control section 150 increases theparallelism of processes for each block if the input image size isrelatively large. This can adaptively prevent delay or data ratedegradation due to a processing amount that increases according to imagesizes. For example, the parallelization control section 150 may controlthe parallelism of processes for each block based on a sequenceparameter set, a picture parameter set, or parameters contained in theslice header. This enables to flexibly configure the parallelismaccording to requirements of users who develop apparatuses. For examplethe parallelism may be configured according to restrictions on theinstallation environment such as the number of processor cores or thenumber of software threads.

The working example can parallelize processes between macro blocks. Thissignifies that any sequence of processes on blocks within an image hasno effect on a finally output result. Accordingly, the parallelizationcontrol section 150 can control a sequence of filtering needdetermination processes in the determination block 110 and a sequence offiltering processes in the horizontal filtering block 130 and thevertical filtering block 140 on a block basis.

More specifically, the parallelization control section 150 may control asequence of filtering processes according to the dependency of thefiltering processes between macro blocks. According to an existingtechnique, for example, the dependency of processes between neighboringmacro blocks around a slice boundary may delay parallel processes oneach slice within an image. However, the parallelization control section150 according to the working example can perform filtering processes onneighboring macro blocks around the slice boundary prior to the othermacro blocks.

For example, FIG. 17 illustrates eight macro blocks MB10 through MB13and MB20 through MB23 around a slice boundary. Macro blocks MB10 throughMB13 belong to slice SL1. Macro blocks MB20 through MB23 belong to sliceSL2. Concerning these macro blocks, the filtering processes forhorizontal boundaries on macro block MB20 in slice SL2 depend on thefiltering processes for vertical boundaries on macro block MB12 in sliceSL1. Similarly, the filtering processes for horizontal boundaries onmacro block MB21 in slice SL2 depend on the filtering processes forvertical boundaries on macro block MB13 in slice SL1.

According to an example in FIG. 18 under these conditions, theparallelization control section 150 performs filtering processes on thevertical boundaries of macro blocks MB12 and MB13 out of filteringprocesses for slice SL1 in preference to processes on the otherboundaries. The result is to prevent a large delay from occurring infiltering processes on the horizontal boundaries of macro blocks MB20and MB21 out of filtering processes for slice SL2. An example in FIG. 19initially performs filtering processes in parallel on verticalboundaries for all macro blocks included in slice SL1. Also in thiscase, no delay occurs in the filtering process on the horizontalboundaries of macro blocks MB20 and MB21 in slice SL2.

[3-2. Process Flow]

With reference to FIGS. 20 through 22, a process flow for the deblockingfilter 24 will be described.

(1) First Scenario

FIG. 20 is a flowchart illustrating a process flow example for thedeblocking filter 24 according to the first scenario. The first scenariocorresponds to the example of large parallelism as shown in FIG. 13.

With reference to FIG. 20, the vertical boundary determination sections112-1 through 112-n determine in parallel whether filtering is neededfor all vertical boundaries included in macro blocks within an inputimage (step S102). The horizontal boundary determination sections 114-1through 114-n determine in parallel whether filtering is needed for allhorizontal boundaries included in macro blocks within an input image(step S104). Steps S102 and S104 are also performed in parallel.

The horizontal filtering sections 132-1 through 132-n apply thedeblocking filter in parallel to all vertical boundaries determined atstep S102 to require the deblocking filter to be applied (step S110).The vertical filtering sections 142-1 through 142-n apply the deblockingfilter in parallel to all horizontal boundaries determined at step S104to require the deblocking filter to be applied (step S120).

(2) Second Scenario

FIG. 21 is a flowchart illustrating a process flow example for thedeblocking filter 24 according to the second scenario. The secondscenario corresponds to the example of smaller parallelism as shown inFIG. 14.

With reference to FIG. 21, the vertical boundary determination sections112-1 through 112-n determine in parallel whether filtering is neededfor all vertical boundaries included in a focused macro block within aninput image (step S202). The horizontal boundary determination sections114-1 through 114-n determine in parallel whether filtering is neededfor all horizontal boundaries included in the focused macro block (stepS204). Steps S202 and S204 are also performed in parallel.

The horizontal filtering sections 132-1 through 132-n apply thedeblocking filter in parallel to vertical boundaries in the focusedmacro block determined at step S202 to require the deblocking filter tobe applied (step S210).

The process at step S220 aims at a focused macro block in the mostrecent loop. The process at step S220 may be skipped for the firstfocused macro block. The vertical filtering sections 142-1 through 142-napply the deblocking filter in parallel to horizontal boundariesdetermined, at step S204 in the most recent loop, to require thedeblocking filter to be applied (step S220).

The process at steps S202 through S220 is repeated for a newly focusedmacro block if focused macro blocks remain unprocessed in the inputimage (step S230).

If there remains no focused macro block unprocessed, the verticalfiltering sections 142-1 through 142-n apply the deblocking filter inparallel to horizontal boundaries determined to require the deblockingfilter to be applied in the focused macro block for the last loop (stepS240). The process then terminates.

While there have been described the two typical scenarios to parallelizeprocesses in units of images and macro blocks, the two scenarios aremere examples for the description. Processes of the deblocking filter 24may be parallelized in various units such as a given number of macroblocks (two or four macro blocks) or a group of horizontally orvertically placed blocks, for example.

(3) Filtering Need Determination Processes

FIG. 22 is a flowchart illustrating a flow of filtering needdetermination processes corresponding to steps S102 and S104 in FIG. 20and steps S202 and S204 in FIG. 21.

With reference to FIG. 22, the distortion evaluation section 124evaluates distortion of each boundary based on the mode information, thetransform coefficient information, and the motion vector information(step S130). The process proceeds to step S134 if the evaluation resultsin the presence of distortion (determination condition A is true). Theprocess proceeds to step S140 if the evaluation results in the absenceof distortion (step S132).

At step S134, the calculation section 122 calculates an edge value basedon a reference pixel tap constituted by the tap constitution section 121(step S134). The threshold comparison section 123 compares thecalculated value with edge determination threshold value β (step S136).The process proceeds to step S138 if the edge value is smaller thanthreshold value β (determination condition B is true). The processproceeds to step S140 if the edge value is not smaller than thresholdvalue β.

At step S138, the filtering determination section 125 determines toapply the deblocking filter to a boundary to be determined (step S138).At step S140, the filtering determination section 125 determines not toapply the deblocking filter to a boundary to be determined (step S140).

4. Second Working Example

The first working example performs the filtering need determinationprocess on a given block using the pixel value of a pixel not updated bythe filtering processes on the other blocks. By contrast, the secondworking example described below provides memory to store pixel valuesinput to the deblocking filter and thereby enables to eliminaterestrictions on the filtering need determination processes and use moreversatile determination conditions.

[4-1. Deblocking Filter Configuration Example]

(1) Description of Sections

FIG. 23 is a block diagram illustrating a detailed configuration of thedeblocking filter 24 according to the second working example. Withreference to FIG. 23, the deblocking filter 24 includes line memory 208,a determination block 210, the horizontal filtering block 130, thevertical filtering block 140, and the parallelization control section150.

The line memory 208 stores pixel values for an input image supplied tothe deblocking filter 24. Filtering processes in the horizontalfiltering block 130 and the vertical filtering block 140 do not updatepixel values stored in the line memory 208. Filtering need determinationprocesses performed by sections described below in the determinationblock 210 reference pixel values stored in the line memory 208. Theapparatus includes another memory for purposes different from processesof the deblocking filter 24. This memory may be reused (shared) as theline memory 208.

The determination block 210 includes vertical boundary determinationsections 212-1 through 212-n and horizontal boundary determinationsections 214-1 through 214-n. The vertical boundary determinationsections 212 and the horizontal boundary determination sections 214 aresupplied with pixel values stored in the line memory 208 for an imageinput to the deblocking filter 24 and determination information used todetermine the need for filtering.

The vertical boundary determination sections 212 use pixel values inputto the deblocking filter 24 to determine whether to apply the deblockingfilter to each vertical boundary. The vertical boundary determinationsections 212 output, to the horizontal filtering block 130, informationindicating a determination result about each vertical boundary.

The horizontal boundary determination sections 214 also use pixel valuesinput to the deblocking filter 24 to determine whether to apply thedeblocking filter to each horizontal boundary. The horizontal boundarydetermination sections 214 perform determination processes in parallelto determination processes performed by the vertical boundarydetermination sections 212. The horizontal boundary determinationsections 214 output, to the vertical filtering block 140, informationindicating a determination result about each horizontal boundary.

(2) Versatile Determination Conditions

Similarly to the existing technique as shown in FIG. 4, the verticalboundary determination sections 212 according to the working example mayreference pixels on the third and sixth rows of a block to determine thenecessity of filtering on a vertical boundary of each block. In thiscase, however, pixel values to be referenced are stored in the linememory 208 and are attributed to an image input to the deblocking filter24. Similarly, the horizontal boundary determination sections 214 mayreference pixels on the third and sixth rows of a block to determine thenecessity of filtering on a horizontal boundary of each block. In thiscase, the configuration according to the working example can be easilyprovided without changing determination conditions for the filteringneed determination processes installed on an existing apparatus.

The vertical boundary determination sections 212 may reference pixels ofthree or more rows in a block during the determination. Similarly, thehorizontal boundary determination sections 214 may reference pixels ofthree or more columns in a block during the determination. The verticalboundary determination sections 212 and the horizontal boundarydetermination sections 214 may use determination condition expressionsdifferent from the existing technique. With reference to FIGS. 24through 26, the following describes six examples of the determinationtechnique according to the working example.

(2-1) First Example

FIG. 24 is an explanatory diagram illustrating first and second examplesof the determination technique. In the first and second examples, thefiltering need determination processes (particularly the determinationusing determination condition B for luma components) for verticalboundaries references pixels of all rows L1 through L8 from the first tothe eighth in each block. The filtering need determination processes forhorizontal boundaries also references pixels of all columns from thefirst to the eighth in each block.

The first example may define determination conditions for lumacomponents as follows.

Determination condition of luma component (Luma) . . . . The deblockingfilter is applied if conditions A and B are both true.

Condition A:

(A1) Block Ba or Bb enters the intra prediction mode;

(A2) Block Ba or Bb has a nonzero orthogonal transform coefficient; or

(A3) |MVAx−MVBx|≥4 or |MVAy−MVBy|≥4

Condition B:

iD₀=|p₂₀−2p₁₀+p₀₀|+|q₂₀−2p₁₀+q₀₀|+|p₂₇−2p₁₇+p₀₇|+|q₂₇−2q₁₇+q₀₇|

iD₁=|p₂₁−2p₁₁+p₀₁|+|q₂₁−2q₁₁+q₀₁|+|p₂₆−2p₁₆+p₀₆|+|q₂₆−2q₁₆+q₀₆|

iD₂=|p₂₂−2p₁₂+p₀₂|+|q₂₂−2q₁₂+q₀₂|+|p₂₅−2p₁₅+p₀₅|+|q₂₅−2q₁₅+q₀₅|

iD₃=|p₂₃−2p₁₃+p₀₃|+|q₂₃−2p₁₃+q₀₃|+|p₂₄−2p₁₄+p₀₄|+|q₂₄−2p₁₄+q₀₄|

iD_(ave)=(iD₀+iD₁+iD₂+iD₃)>>2

Under this condition, iD_(ave)<β

The determination condition for chroma components may be equal to theabove-described existing technique. A weighted average may be calculatedto calculate average iD_(ave) for four determination parameters iD₀through iD₃.

(2-2) Second Example

The second example may define determination condition B for lumacomponents as follows.

Condition B:

iD₀=|p₂₀−2p₁₀+p₀₀|+|q₂₀−2q₁₀+q₀₀|+|p₂₇−2p₁₇+p₀₇|+|q₂₇−2q₁₇+q₀₇|

iD₁=|p₂₁−2p₁₁+p₀₁|+|q₂₁−2q₁₁+q₀₁|+|p₂₆−2p₁₆+p₀₆|+|q₂₆−2q₁₆+q₀₆|

iD₂=|p₂₂−2p₁₂+p₀₂|+|q₂₂−2q₁₂+q₀₂|+|p₂₅−2p₁₅+p₀₅|+|q₂₅−2q₁₅+q₀₅|

iD₃=|p₂₃−2p₁₃+p₀₃|+|q₂₃−2q₁₃+q₀₃|+|p₂₄−2p₁₄+p₀₄|+|q₂₄−2q₁₄+q₀₄|

Under this condition, iD₀<β and iD₁<β and iD₂<β and iD₃<β

An equation to calculate four determination parameters iD₀ through iD₃is equal to that of the first example. An available condition is thatnot all of, but at least three, two, or one of four determinationparameters iD₀ through iD₃ is smaller than edge determination thresholdvalue β.

(2-3) Third Example

FIG. 25 is an explanatory diagram illustrating third and fourth examplesof the determination technique. In the third and fourth examples, thefiltering need determination processes (particularly the determinationusing determination condition B for luma components) for verticalboundaries references pixels of four rows L1, L3, L6, and L8 in eachblock. The filtering need determination processes for horizontalboundaries also references pixels of four columns in each block.

The third example may define determination conditions for lumacomponents as follows.

Determination condition of luma component (Luma) . . . . The deblockingfilter is applied if conditions A and B are both true.

Condition A:

(A1) Block Ba or Bb enters the intra prediction mode;

(A2) Block Ba or Bb has a nonzero orthogonal transform coefficient; or

(A3) |MVAx−MVBx|≥4 or |MVAy−MVBy|≥4

Condition B:

iD₀=|p₂₀−2p₁₀+p₀₀|+|q₂₀−2q₁₀+q₀₀|+|p₂₇−2p₁₇+p₀₇|+|q₂₇−2q₁₇+q₀₇|

iD₂=|p₂₂−2p₁₂+p₀₂|+|q₂₂−2q₁₂+q₀₂|+|p₂₅−2p₁₅+p₀₅|+|q₂₅−2q₁₅+q₀₅|

iD_(ave)=(iD₀+iD₂)>>1

Under this condition, iD_(ave)<β

The determination condition for chroma components may be equal to theabove-described existing technique. A weighted average may be calculatedto calculate average iD_(ave) for two determination parameters iD₀ andiD₂.

(2-4) Fourth Example

The fourth example may define determination condition B for lumacomponents as follows.

Condition B:

iD₀=|p₂₀−2p₁₀+p₀₀|+|q₂₀−2q₁₀+q₀₀|+|q₂₇−2p₁₇+p₀₇|+|q₂₇−2q₁₇+q₀₇|

iD₂=|p₂₂−2p₁₂+p₀₂|+|q₂₂−2q₁₂+q₀₂|+|p₂₅−2p₁₅+p₀₅|+|q₂₅−2q₁₅+q₀₅|

Under this condition, iD₀<β and iD₂<β

An equation to calculate two determination parameters iD₀ and iD₂ isequal to that of the third example. An available condition is that notboth of, but either of two determination parameters iD₀ and iD₂ issmaller than edge determination threshold value β.

While there has been described the example of referencing the first,third, sixth, and eighth rows (or columns) L1, L3, L6, and L8 in a blockduring the determination, the other combinations of rows or columns maybe referenced.

(2-5) Fifth Example

FIG. 26 is an explanatory diagram illustrating fifth and sixth examplesof the determination technique. In the fifth and sixth examples, thefiltering need determination processes for vertical boundariesreferences pixels of four rows L1, L3, L5, and L7 in each block. Thefiltering need determination processes for horizontal boundaries alsoreferences pixels of four columns in each block.

The fifth example may define determination conditions for lumacomponents as follows.

Determination condition of luma component (Luma) . . . . The deblockingfilter is applied if conditions A and B are both true.

Condition A:

(A1) Block Ba or Bb enters the intra prediction mode;

(A2) Block Ba or Bb has a nonzero orthogonal transform coefficient; or

(A3) |MVAx−MVBx|≥4 or |MVAy−MVBy|≥4

Condition B:

iD₀=|p₂₀−2p₁₀+p₀₀|+|q₂₀−2q₁₀+q₀₀|+|q₂₆−2p₁₆+p₀₆|+|q₂₆−2q₁₆+q₀₆|

iD₂=|p₂₂−2p₁₂+p₀₂|+|q₂₂−2q₁₂+q₀₂|+|p₂₄−2p₁₄+p₀₄|+|q₂₄−2q₁₄+q₀₄|

iD_(ave)=(iD₀+iD₂)>>1

Under this condition, iD_(ave)<β

The determination condition for chroma components may be equal to theabove-described existing technique. A weighted average may be calculatedto calculate average iD_(ave) for two determination parameters iD₀ andiD₂.

(2-6) Sixth Example

The sixth example may define determination condition B for lumacomponents as follows.

Condition B:

iD₀=|p₂₀−2p₁₀+p₀₀|+|q₂₀−2q₁₀+q₀₀|+|p₂₆−2p₁₆+p₀₆|+|q₂₆−2q₁₆+q₀₆|

iD₂=|p₂₂−2p₁₂+p₀₂|+|q₂₂−2q₁₂+q₀₂|+|p₂₄−2p₁₄+p₀₄|+|q₂₄−2q₁₄+q₀₄|

Under this condition, iD₀<β and iD₂<β

An equation to calculate two determination parameters iD₀ and iD₂ isequal to that of the fifth example. An available condition is that notboth of, but either of two determination parameters iD₀ and iD₂ issmaller than edge determination threshold value β.

Generally, increasing the number of rows and columns to be referencedfor the determination improves the determination accuracy. Therefore,the first and second examples of referencing eight rows and columns canminimize a possibility of filtering a block originally not targeted forthe deblocking filter to be applied and a possibility of not filtering ablock originally targeted for the deblocking filter to be applied. Theresult is to improve the quality of an image to be encoded and decode.On the other hand, decreasing the number of rows and columns to bereferenced for the determination can reduce processing costs. Sincethere is trade-off between the image quality and the processing cost, itmay be advantageous to adaptively select the number of rows and columnsto be referenced for the determination depending on the use of theapparatus or restrictions on the installation. It may be advantageous toadaptively select combinations of rows and columns to be referenced.

As described in the first, third, and fifth examples, average valueiD_(ave) of determination parameters can be compared with edgedetermination threshold value β to appropriately perform thedetermination on a block basis without an excess effect of parametervariations for each row or column.

[4-2. Process Flow]

According to the second working example as well as the first workingexample, the deblocking filter 24 can operate on different parallelisms.

In the first scenario using a large parallelism, the vertical boundarydetermination sections 212-1 through 212-n determine in parallel whetherthe filtering is needed for all vertical boundaries included in macroblocks within an input image. The horizontal boundary determinationsections 214-1 through 214-n determine in parallel whether filtering isneeded for all horizontal boundaries included in macro blocks within aninput image. The determination processes for vertical boundaries andhorizontal boundaries are also performed in parallel. The horizontalfiltering sections 132-1 through 132-n and the vertical filteringsections 142-1 through 142-n apply the deblocking filter to a verticalboundary and a horizontal boundary determined to require the deblockingfilter to be applied (see FIG. 20).

In the second scenario using a small parallelism, the vertical boundarydetermination sections 212-1 through 212-n determine in parallel whetherthe filtering is needed for all vertical boundaries included in onefocused macro block within an input image. The horizontal boundarydetermination sections 214-1 through 214-n determine in parallel whetherfiltering is needed for all horizontal boundaries included in thefocused macro block. The determination processes for vertical boundariesand horizontal boundaries are also performed in parallel. The horizontalfiltering sections 132-1 through 132-n then apply the deblocking filterin parallel to a vertical boundary determined to require the deblockingfilter to be applied. The vertical filtering sections 142-1 through142-n apply the deblocking filter in parallel to a horizontal boundarydetermined to require the deblocking filter to be applied. This processis repeatedly performed on all macro blocks within an input image (seeFIG. 21).

These two scenarios are mere examples for the description. Also in thesecond working example, processes of the deblocking filter 24 may beparallelized in various units such as a given number of macro blocks(two or four macro blocks) or a group of horizontally or verticallyplaced blocks, for example. Also in the second working example as wellas the first working example, the parallelization control section 150may control the parallelism and sequence of processes in the deblockingfilter 24.

[4-3. Process Example for Each LCU]

As already mentioned, the technology according to various workingexamples described in this specification may be provided as a processbased on an HEVC encoding unit (CU). According to HEVC, an encoding unithaving the largest size is referred to as a largest coding unit (LCU)that can be selected as 64×64 pixels, for example. The minimumselectable CU size is 8×8 pixels. Normally, an image is encoded anddecoded corresponding to each LCU in accordance with a raster scansequence from the LCU at the top left of a picture (or a slice). Thefollowing describes process examples corresponding to LCUs in thedeblocking filter 24.

FIG. 27 is an explanatory diagram illustrating a process sequence foreach LCU according to the second working example described above. Theexample assumes the LCU size to be 16×16 pixels and the CU size to be8×8 pixels.

With reference to FIG. 27, the first stage is shown at the top left ofthe drawing and indicates that the filtering on LCUs has completed up tothe (n−1)th LCU. Shaded pixels are targeted for filtering on verticalboundaries. Filled pixels are targeted for filtering on horizontalboundaries.

Processes for the second stage at the top right of FIG. 27 and the thirdstage at the bottom left thereof are targeted for the nth LCU. Beforethe second stage, pixel values input to the deblocking filter 24 areused to perform the filtering need determination processes in parallelon all vertical boundaries and horizontal boundaries belonging to thenth LCU. At the second stage, pixel values input to the deblockingfilter 24 are used to perform the filtering process in parallel on avertical boundary that belongs to the nth LCU and is determined torequire the deblocking filter to be applied. The pixel values processedat the second stage are then used to perform the filtering process inparallel on a horizontal boundary that belongs to the nth LCU and isdetermined to require the deblocking filter to be applied.

A process for the fourth stage at the bottom right of FIG. 27 istargeted for the (n+1)th LCU. At the fourth stage, the filtering processis performed in parallel on a vertical boundary determined to requirethe deblocking filter to be applied after the filtering needdetermination processes are performed in parallel on all verticalboundaries and horizontal boundaries belonging to the (n+1)th LCU.

While the example assumes the LCU size to be 16×16 pixels, it may be setto 32×32 or 64×64 pixels. The effect of shortening the processing timeaccording to the parallelization is further improved because increasingthe size of an LCU to be selected also increases the number of verticalboundaries and horizontal boundaries belonging to one LCU.

FIG. 28 is a flowchart illustrating a process flow of the deblockingfilter 24 for each LCU.

With reference to FIG. 28, the vertical boundary determination sections212-1 through 212-n determine in parallel whether filtering is neededfor all vertical boundaries included in a focused LCU within an inputimage (step S252). The horizontal boundary determination sections 214-1through 214-n determine in parallel whether filtering is needed for allhorizontal boundaries included in the focused LCU (step S254). StepsS252 and S254 are also performed in parallel.

The horizontal filtering sections 132-1 through 132-n apply thedeblocking filter in parallel to vertical boundaries in the focused LCUdetermined at step S252 to require the deblocking filter to be applied(step S260).

The vertical filtering sections 142-1 through 142-n apply the deblockingfilter in parallel to horizontal boundaries in the focused LCUdetermined at step S254 to require the deblocking filter to be applied(step S270).

The process at steps S252 through S270 is repeated for a newly focusedLCU if an LCU remains unprocessed in the input image (step S280). Theprocess terminates if there remains no LCU unprocessed.

5. Third Working Example

[5-1. Overview]

According to the second working example, the filtering needdetermination processes on vertical boundaries and horizontal boundariesreference a pixel value input to the deblocking filter and therebyeliminating the dependency between the filtering need determinationprocesses and making the parallelization of determination processespossible. The third working example described below applies the conceptof the second working example also to the filtering process. Thefiltering processes for vertical boundaries and horizontal boundariesalso filter pixel values input to the deblocking filter.

FIG. 29 is an explanatory diagram illustrating an overview of theworking example. At the bottom left of FIG. 29, there is shown a shaperepresenting input pixels (also referred to as reconstruct pixels)before being processed by the deblocking filter. The working exampleallows pixel values input to the deblocking filter to be referenced fromnot only filtering need determination processes for vertical boundariesand horizontal boundaries but also filtering processes for verticalboundaries and filtering processes for horizontal boundaries. Therefore,the dependency between the two filtering processes is eliminated. Thetwo filtering processes are performed in parallel.

The filtering processes for vertical boundaries and the filteringprocesses for horizontal boundaries may update values of duplicatepixels. Filled pixels in FIG. 29 illustrate positions of the pixelslikely to be duplicated. The deblocking filter according to the workingexample calculates one output pixel value from two filter outputs interms of pixels that are duplicately updated by two filters operating inparallel.

[5-2. Deblocking Filter Configuration Example]

FIG. 30 is a block diagram illustrating a detailed configuration of thedeblocking filter 24 according to the third working example. Withreference to FIG. 30, the deblocking filter 24 includes the line memory208, the determination block 210, a horizontal filtering block 330, avertical filtering block 340, the parallelization control section 150, acalculation section 360. The determination block 210 includes verticalboundary determination sections 212-1 through 212-n and horizontalboundary determination sections 214-1 through 214-n. The verticalboundary determination sections 212 and the horizontal boundarydetermination sections 214 may determine the necessity of filtering onboundaries according to the various determination conditions asdescribed in the second working example.

The horizontal filtering block 330 includes horizontal filteringsections 332-1 through 332-n. The horizontal filtering sections 332 aresupplied with an input image value from the line memory 208 and adetermination result concerning each vertical boundary from thedetermination block 210.

The horizontal filtering sections 332 apply the deblocking filter forvertical boundaries to right and left pixels around the correspondingvertical boundary if the determination result from the vertical boundarydetermination section 212 indicates that the filter needs to be applied.The horizontal filtering sections 332 output, to the calculation section360, a pixel value after the filtering in terms of the filtered pixel oran input pixel value in terms of the other pixels.

The vertical filtering block 340 includes vertical filtering sections342-1 through 342-n. The vertical filtering sections 342 are suppliedwith an input pixel value from the line memory 208 and a determinationresult concerning each horizontal boundary from the determination block210.

The vertical filtering sections 342 apply the deblocking filter forhorizontal boundaries to top and bottom pixels around the correspondinghorizontal boundary if the determination result from the horizontalboundary determination section 214 indicates that the filter needs to beapplied. Filtering processes of the vertical filtering sections 342-1through 342-n are performed in parallel to filtering processes of thehorizontal filtering sections 332-1 through 332-n. The verticalfiltering sections 342 output, to the calculation section 360, a pixelvalue after the filtering in terms of the filtered pixel or an inputpixel value in terms of the other pixels.

The calculation section 360 is supplied with an output pixel value fromthe horizontal filtering block 330 and an output pixel value from thevertical filtering block 340 in parallel. Further, the calculationsection 360 is supplied with determination results from the verticalboundary determination section 212 and the horizontal boundarydetermination section 214. According to a determination result, thecalculation section 360 calculates output pixel values for pixelsfiltered from the horizontal filtering block 330 and the verticalfiltering block 340 based on filter outputs from the horizontalfiltering block 330 and the vertical filtering block 340.

According to the working example, for example, the calculation section360 calculates an average of two filter outputs for duplicately filteredpixels. The calculation section 360 may calculate a simple average oftwo filter outputs. Instead, the calculation section 360 may calculate aweighted average of two filter outputs. For example, the calculationsection 360 may determine a weight for weighted averages of pixelsaccording to the distance from each pixel to the vertical boundary andto the horizontal boundary.

FIG. 31 is an explanatory diagram illustrating determination of a weightfor weighted average calculated by the calculation section 360. FIG. 31shows focused pixel P_(Z) in black corresponding to one of theduplicated positions illustrated in FIG. 29. There are three pixelscorresponding to distance D_(V) between focused pixel P_(Z) and nearestvertical boundary V_(Z). There are two pixels corresponding to distanceD_(H) between focused pixel P_(Z) and nearest horizontal boundary H_(Z).Distance D_(H) is smaller than distance D_(V). In this case, thecalculation section 360 may set a weight for output from the deblockingfilter applied to horizontal boundary H_(Z) to be larger than a weightfor output from the deblocking filter applied to vertical boundaryV_(Z). The example in FIG. 31 assumes that a ratio of filter outputV_(out) for vertical boundary V_(Z) to filter output H_(out) forhorizontal boundary H_(Z) is 2:3.

As seen from FIG. 31, calculating a weighted average of two filteroutputs can consequently provide each focused pixel with an output pixelvalue similar to the case of applying one two-dimensional filter havinga filter tap along the horizontal direction and a filter tap along thevertical direction. Parallelizing filtering processes on the verticalboundary and the horizontal boundary can also appropriately reduce blockdistortion appearing on the vertical boundary and the horizontalboundary. As another working example, the deblocking filter 24 mayinclude one two-dimensional filter that simultaneously calculateshorizontal filtering, vertical filtering, and a weighted average. Inthis case, however, the installation is very complicated because filtercoefficients need to be variously changed correspondingly to pixels. Onthe other hand, the third working example performs two one-dimensionalfilters in parallel and then calculates a weighted average. This caneasily provide processes substantially equal to a two-dimensional filterwhile ensuring the functionality of existing deblocking filters.

FIG. 32 is an explanatory diagram illustrating an example of the weightfor weighted averages calculated based on the example in FIG. 31. FIG.32 shows 36 pixels (6×6) around an intersection between the verticalboundary and the horizontal boundary. The pixels correspond to theabove-described duplicated positions. The ratio of the weight for filteroutput V_(out) to the weight for filter output H_(out) is 1:1 (2:2 or3:3) for pixels positioned at an equal distance from the verticalboundary and the horizontal boundary. The weight for filter outputV_(out) is larger than the weight for filter output H_(out) for pixelsnearer to the vertical boundary. For example, the ratio of weights forpixel P₁ is V_(out):H_(out)=3:1. The weight for filter output V_(out) issmaller than the weight for filter output H_(out) for pixels nearer tothe horizontal boundary. For example, the ratio of weights for pixel P₂is V_(out):H_(out)=1:3.

The block distortion can be more effectively suppressed and the imagequality can be improved by varying the weight for weighted averagesdepending on the distance between each pixel and the boundary.

The above-described weights are mere examples. For example, thecalculation section 360 may determine the weight of weighted averagesfor pixels according to the edge strengths of the vertical boundary andthe horizontal boundary corresponding to each pixel instead of or inaddition to the distance between each pixel and the boundary. The edgestrength may be represented with a parameter such as an edge valuecalculated from the calculation section 122 as shown in FIG. 16, forexample. In this case, the weight for filer output on a boundary havinga stronger edge may be set to be larger than the weight for filer outputon a boundary having a weaker edge. Varying the weight of weightedaverages according to the edge strength can adaptively improve theeffect of the deblocking filter at a boundary that remarkably causesblock distortion.

The calculation section 360 selects outputs from actually filteredblocks in terms of pixels filtered by one of the horizontal filteringblock 330 and the vertical filtering block 340. The calculation section360 directly outputs an input pixel value to be output to the deblockingfilter 24 in terms of pixels not filtered by the horizontal filteringblock 330 or the vertical filtering block 340. A table in FIG. 33 listsoutput pixel values from the calculation section 360 according toresults of the determination whether to require the filtering.

[5-3. Process Sequence Example]

The following describes two examples of process sequences available forthe deblocking filter 24 according to the working example. The examplealso assumes that the deblocking filter is supplied with an image havingthe size of 32×32 pixels. The input image includes four macro blocks MB0through MB3 each having the size of 16×16 pixels.

(1) First Example

For comparison, FIG. 34 illustrates a process sequence when thedependency remains between a filtering process on the vertical boundaryand a filtering process on the horizontal boundary. The process sequencein FIG. 34 substantially equals the process sequence in FIG. 13according to the first working example. In FIG. 34, the first stepperforms, in parallel, filtering need determination processes J_(V0,0)through J_(V3,3) and J_(H0,0) through J_(H3,3) on all verticalboundaries and all horizontal boundaries of all the four macro blocksMB0 through MB3. The second step performs filtering processes F_(V0,0)through F_(V3,3) on 16 vertical boundaries of the four macro blocks MB0through MB3. The third step performs filtering processes F_(H0,0)through F_(H3,3) on 16 horizontal boundaries of the four macro blocksMB0 through MB3. The fourth step (omitted from FIG. 13) stores pixelvalues after the filtering process on the horizontal boundary in thememory used for outputs from the deblocking filter 24.

FIG. 35 illustrates a first example of process sequence provided by theworking example. In FIG. 35, the first step performs, in parallel,filtering need determination processes J_(V0,0) through J_(V3,3) andJ_(H0,0) through J_(H3,3) on all vertical boundaries and all horizontalboundaries of the four macro blocks MB0 through MB3. The second stepperforms, in parallel, filtering processes F_(V0,0) through F_(V3,3) andF_(H0,0) through F_(H3,3) on all vertical boundaries and all horizontalboundaries of the four macro blocks MB0 through MB3. Actually, thesecond step filters only a boundary determined to require the filtering.The third step stores pixel values in the memory used for outputs fromthe deblocking filter 24. A weighted average of two filter outputs maybe calculated as an output pixel value in terms of pixels filtered bythe horizontal filtering block 330 and the vertical filtering block 340.

(2) Second Example

While the first example maximizes the parallelism, the deblocking filter24 according to the second example can also perform a process for eachmacro block.

For comparison, FIG. 36 illustrates a process sequence for each macroblock when the dependency remains between a filtering process on thevertical boundary and a filtering process on the horizontal boundary.The process sequence in FIG. 36 substantially equals the processsequence in FIG. 14 according to the first working example. FIG. 36explicitly shows the four process steps (sixth, tenth, 14th, and 16th)that store pixel values in the memory for output and are omitted fromFIG. 14 for simplicity. Sixteen process steps including the four processsteps configure the process in FIG. 36.

FIG. 37 illustrates a second example of process sequence provided by theworking example. In FIG. 37, the first step performs, in parallel,filtering need determination processes J_(V0,0) through J_(V0,3) andJ_(H0,0) through J_(H0,3) on four vertical boundaries and fourhorizontal boundaries of macro block MB0. The second step performs, inparallel, filtering processes F_(V0,0) through F_(V0,3) and F_(H0,0)through F_(H0,3) on four vertical boundaries and four horizontalboundaries of macro block MB0. The third step stores pixel values ofmacro block MB0 in the memory used for outputs from the deblockingfilter 24. A weighted average of two filter outputs may be calculated asan output pixel value in terms of pixels duplicately filtered by twofilters. The fourth to sixth steps similarly process macro block MB1.The seventh to ninth steps similarly process macro block MB2. The tenthto twelfth steps similarly process macro block MB3. The process in FIG.37 includes twelve process steps fewer than those of the process in FIG.36.

The third working example eliminates the dependency between filteringprocesses for vertical boundaries and a filtering process for horizontalboundaries. The process of the deblocking filter 24 can be performedusing fewer process steps than those used for the first and secondworking examples. One of advantages of allowing a filtering process toreference only pixels input to the deblocking filter is that anyconfiguration of filter taps causes no dependency between filteringprocesses for vertical boundaries and filtering processes for horizontalboundaries. The third working example can improve the image quality byconfiguring a filter tap using more pixels than used for existingtechniques. For example, the existing technique uses a filter tap ofthree pixels for each side of each boundary as described with referenceto FIG. 7. The working example causes no dependency between processeseven if a filter tap of five pixels or more is used at each boundary. Nodependency occurs between processes even by further decreasing the blocksize as a process unit for the deblocking filter.

Also in the third working example as well as the first and secondworking examples, the parallelization control section 150 may controlthe parallelism and sequence of processes in the deblocking filter 24.

[5-4. Process Flow]

FIG. 38 is a flowchart illustrating an example of a process flow for thedeblocking filter according to the third working example. FIG. 39 is aflowchart illustrating a flow of the pixel value calculation processshown in FIG. 38.

With reference to FIG. 38, the vertical boundary determination sections212-1 through 212-n determine in parallel whether filtering is neededfor all vertical boundaries within an input image or a macro block (stepS302). The horizontal boundary determination sections 214-1 through214-n determine in parallel whether filtering is needed for allhorizontal boundaries within the input image or the macro block (stepS304). Steps S302 and S304 are also performed in parallel.

The horizontal filtering sections 332-1 through 332-n apply thedeblocking filter in parallel to all vertical boundaries determined atstep S302 to require the deblocking filter to be applied (step S306).The vertical filtering sections 342-1 through 342-n apply the deblockingfilter in parallel to all horizontal boundaries determined at step S304to require the deblocking filter to be applied (step S308). Steps S306and S308 are also performed in parallel.

The calculation section 360 then performs the pixel value calculationprocess as shown in FIG. 39 (step S310). With reference to FIG. 39, theprocess from step S314 to step S326 loops for each pixel to be processed(step S312).

At step S314, the calculation section 360 determines whether two filtersfor vertical boundaries and horizontal boundaries have filtered afocused pixel (step S314). The process proceeds to step S322 if the twofilters have filtered the focused pixel. The process proceeds to stepS316 if the two filters have not filtered the focused pixel.

At step S316, the calculation section 360 determines whether one of thetwo filters for vertical boundaries and horizontal boundaries hasfiltered the focused pixel (step S316). The process proceeds to stepS320 if one of the two filters has filtered the focused pixel. Theprocess proceeds to step S318 if none of the filters has filtered thefocused pixel.

At step S318, the calculation section 360 acquires an input pixel valueto the deblocking filter 24 (step S318). At step S320, the calculationsection 360 acquires a filter output from the filter that actuallyfilters the focused pixel (step S320).

At step S322, the calculation section 360 determines weight values forcalculating a weighted average of filter outputs from the two filtersconcerning the focused pixel according to distances from the focusedpixel to the vertical boundary and the horizontal boundary or the edgestrengths of the vertical boundary and the horizontal boundarycorresponding to the focused pixel (step S322). The calculation section360 calculates a weighted average of filter outputs from the two filtersusing the determined weight (step S324).

The calculation section 360 stores the pixel value of the focused pixelin the memory while the pixel value is acquired at step S318 or S320 oris calculated at step S324 (step S326). The sequences of processes asshown in FIGS. 38 and 39 terminate when the process is performed on allpixels to be processed.

6. Application to Various Codecs

The technology according to the disclosure is applicable to variouscodecs related to image encoding and decoding. The following describesexamples of applying the technology according to the disclosure tomultiview codec and scalable codec.

[6-1. Multiview Codec]

The multiview codec is an image encoding system that encodes and decodesmultiple-perspective video. FIG. 40 is an explanatory diagramillustrating the multiview codec. FIG. 40 illustrates sequences offrames for three views captured at three observing points. Each view isprovided with a view ID (view_id). One of the views is specified as abase view. Views other than the base view are referred to as non-baseviews. The example in FIG. 40 represents a base view with view ID “0”and two non-base views with view ID “1” or “2.” Encoding multiview imagedata may compress the data size of the encoded stream as a whole byencoding frames of the non-base view based on encoding information aboutframes of the base view.

The deblocking filter may be applied to each view during the encodingprocess and the decoding process according to the multiview codecdescribed above. When the deblocking filter is applied to each view,filtering need determination processes on vertical block boundaries andhorizontal block boundaries for each view may be parallelized based onthe technology according to the disclosure. A filtering needdetermination process and a filtering process for each view may beparallelized. The parameter (such as the one described in the precedingparagraph 0095) to control a filtering need determination process or afiltering process may be provided for each view. A parameter providedfor the base view may be reused for the non-base view.

Filtering need determination processes or filtering processes may beparallelized over views. Views may share the parameter (such as the onedescribed in the preceding paragraph 0095) to control filtering needdetermination processes or filtering processes. It may be advantageousto additionally specify a flag indicating whether views share theparameter.

FIG. 41 is an explanatory diagram illustrating an image encoding processapplied to the multiview codec described above. FIG. 41 shows aconfiguration of a multiview encoding device 710 as an example. Themultiview encoding device 710 includes a first encoding section 720, asecond encoding section 730, and a multiplexing section 740.

The first encoding section 720 encodes a base view image and generatesan encoded stream for the base view. The second encoding section 730encodes a non-base view image and generates an encoded stream for thenon-base view. The multiplexing section 740 multiplexes an encodedstream for the base view generated from the first encoding section 720and one or more encoded streams for the non-base view generated from thesecond encoding section 730 to generate a multiplexed stream formultiview.

The first encoding section 720 and the second encoding section 730illustrated in FIG. 41 are configured similarly to the image encodingdevice 10 according to the above-described embodiment. Applying thedeblocking filter to views enables to parallelize filtering needdetermination processes for vertical block boundaries and horizontalblock boundaries or parallelize filtering need determination processesand filtering processes. A parameter to control these processes may beinserted into a header area of the encoded stream for each view or intoa common header area in the multiplexed stream.

FIG. 42 is an explanatory diagram illustrating an image decoding processapplied to the multiview codec described above. FIG. 42 shows aconfiguration of a multiview decoding device 760 as an example. Themultiview decoding device 760 includes a demultiplexing section 770, afirst decoding section 780, and a second decoding section 790.

The demultiplexing section 770 demultiplexes a multiplexed stream formultiview into an encoded stream for the base view and an encoded streamfor one or more non-base views. The first decoding section 780 decodes abase view image from an encoded stream for the base view. The seconddecoding section 790 decodes a non-base view image from an encodedstream for the non-base view.

The first decoding section 780 and the second decoding section 790illustrated in FIG. 42 are configured similarly to the image decodingdevice 60 according to the above-described embodiment. Applying thedeblocking filter to views enables to parallelize filtering needdetermination processes for vertical block boundaries and horizontalblock boundaries or parallelize filtering need determination processesand filtering processes. A parameter to control these processes may beacquired from a header area of the encoded stream for each view or froma common header area in the multiplexed stream.

[6-2. Scalable Codec]

The scalable codec is an image encoding system to provide hierarchicalencoding. FIG. 43 is an explanatory diagram illustrating the scalablecodec. FIG. 43 illustrates frame sequences for three layers of differentspace resolutions, time resolutions, or image qualities. Each layer isprovided with a layer ID (layer id). These layers include a base layerhaving the lowest resolution (or image quality). Layers other than thebase layer are referred to as enhancement layers. The example in FIG. 43represents a base layer with layer ID “0” and two enhancement layerswith layer ID “1” or “2.” Encoding multi-layer image data may compressthe data size of the encoded stream as a whole by encoding frames of theenhancement layer based on encoding information about frames of the baselayer.

The deblocking filter may be applied to each layer during the encodingprocess and the decoding process according to the scalable codecdescribed above. When the deblocking filter is applied to each view,filtering need determination processes on vertical block boundaries andhorizontal block boundaries for each layer may be parallelized based onthe technology according to the disclosure. A filtering needdetermination process and a filtering process for each layer may beparallelized. The parameter (such as the one described in the precedingparagraph 0095) to control a filtering need determination process or afiltering process may be provided for each layer. A parameter providedfor the base layer may be reused for the enhancement layer.

Filtering need determination processes or filtering processes may beparallelized over layers. Layers may share the parameter (such as theone described in the preceding paragraph 0095) to control filtering needdetermination processes or filtering processes. It may be advantageousto additionally specify a flag indicating whether layers share theparameter.

FIG. 44 is an explanatory diagram illustrating an image encoding processapplied to the scalable codec described above. FIG. 44 shows aconfiguration of a scalable encoding device 810 as an example. Thescalable encoding device 810 includes a first encoding section 820, asecond encoding section 830, and a multiplexing section 840.

The first encoding section 820 encodes a base layer image and generatesan encoded stream for the base layer. The second encoding section 830encodes an enhancement layer image and generates an encoded stream forthe enhancement layer.

The multiplexing section 840 multiplexes an encoded stream for the baselayer generated from the first encoding section 820 and one or moreencoded streams for the enhancement layer generated from the secondencoding section 830 to generate a multiplexed stream for multi-layer.

The first encoding section 820 and the second encoding section 830illustrated in FIG. 44 are configured similarly to the image encodingdevice 10 according to the above-described embodiment. Applying thedeblocking filter to layers enables to parallelize filtering needdetermination processes for vertical block boundaries and horizontalblock boundaries or parallelize filtering need determination processesand filtering processes. A parameter to control these processes may beinserted into a header area of the encoded stream for each layer or intoa common header area in the multiplexed stream.

FIG. 45 is an explanatory diagram illustrating an image decoding processapplied to the scalable codec described above. FIG. 45 shows aconfiguration of a scalable decoding device 860 as an example. Thescalable decoding device 860 includes a demultiplexing section 870, afirst decoding section 880, and a second decoding section 890.

The demultiplexing section 870 demultiplexes a multiplexed stream formulti-layer into an encoded stream for the base layer and an encodedstream for one or more enhancement layers. The first decoding section880 decodes a base layer image from an encoded stream for the baselayer. The second decoding section 890 decodes an enhancement layerimage from an encoded stream for the enhancement layer.

The first decoding section 880 and the second decoding section 890illustrated in FIG. 45 are configured similarly to the image decodingdevice 60 according to the above-described embodiment. Applying thedeblocking filter to layers enables to parallelize filtering needdetermination processes for vertical block boundaries and horizontalblock boundaries or parallelize filtering need determination processesand filtering processes. A parameter to control these processes may beacquired from a header area of the encoded stream for each layer or froma common header area in the multiplexed stream.

7. Example Application

The image encoding device 10 and the image decoding device 60 accordingto the embodiment described above may be applied to various electronicappliances such as a transmitter and a receiver for satellitebroadcasting, cable broadcasting such as cable TV, distribution on theInternet, distribution to terminals via cellular communication, and thelike, a recording device that records images in a medium such as anoptical disc, a magnetic disk or a flash memory, a reproduction devicethat reproduces images from such storage medium, and the like. Fourexample applications will be described below.

7-1. First Example Application

FIG. 46 is a block diagram showing an example of a schematicconfiguration of a television adopting the embodiment described above. Atelevision 900 includes an antenna 901, a tuner 902, a demultiplexer903, a decoder 904, an video signal processing section 905, a displaysection 906, an audio signal processing section 907, a speaker 908, anexternal interface 909, a control section 910, a user interface 911, anda bus 912.

The tuner 902 extracts a signal of a desired channel from broadcastsignals received via the antenna 901, and demodulates the extractedsignal. Then, the tuner 902 outputs an encoded bit stream obtained bydemodulation to the demultiplexer 903. That is, the tuner 902 serves astransmission means of the televisions 900 for receiving an encodedstream in which an image is encoded.

The demultiplexer 903 separates a video stream and an audio stream of aprogram to be viewed from the encoded bit stream, and outputs eachstream which has been separated to the decoder 904. Also, thedemultiplexer 903 extracts auxiliary data such as an EPG (ElectronicProgram Guide) from the encoded bit stream, and supplies the extracteddata to the control section 910. Additionally, the demultiplexer 903 mayperform descrambling in the case the encoded bit stream is scrambled.

The decoder 904 decodes the video stream and the audio stream input fromthe demultiplexer 903. Then, the decoder 904 outputs video datagenerated by the decoding process to the video signal processing section905. Also, the decoder 904 outputs the audio data generated by thedecoding process to the audio signal processing section 907.

The video signal processing section 905 reproduces the video data inputfrom the decoder 904, and causes the display section 906 to display thevideo. The video signal processing section 905 may also cause thedisplay section 906 to display an application screen supplied via anetwork. Further, the video signal processing section 905 may perform anadditional process such as noise removal, for example, on the video dataaccording to the setting. Furthermore, the video signal processingsection 905 may generate an image of a GUI (Graphical User Interface)such as a menu, a button, a cursor or the like, for example, andsuperimpose the generated image on an output image.

The display section 906 is driven by a drive signal supplied by thevideo signal processing section 905, and displays a video or an image onan video screen of a display device (for example, a liquid crystaldisplay, a plasma display, an OLED, or the like).

The audio signal processing section 907 performs reproduction processessuch as D/A conversion and amplification on the audio data input fromthe decoder 904, and outputs audio from the speaker 908. Also, the audiosignal processing section 907 may perform an additional process such asnoise removal on the audio data.

The external interface 909 is an interface for connecting the television900 and an external appliance or a network. For example, a video streamor an audio stream received via the external interface 909 may bedecoded by the decoder 904. That is, the external interface 909 alsoserves as transmission means of the televisions 900 for receiving anencoded stream in which an image is encoded.

The control section 910 includes a processor such as a CPU (CentralProcessing Unit), and a memory such as an RAM (Random Access Memory), anROM (Read Only Memory), or the like. The memory stores a program to beexecuted by the CPU, program data, EPG data, data acquired via anetwork, and the like. The program stored in the memory is read andexecuted by the CPU at the time of activation of the television 900, forexample. The CPU controls the operation of the television 900 accordingto an operation signal input from the user interface 911, for example,by executing the program.

The user interface 911 is connected to the control section 910. The userinterface 911 includes a button and a switch used by a user to operatethe television 900, and a receiving section for a remote control signal,for example. The user interface 911 detects an operation of a user viathese structural elements, generates an operation signal, and outputsthe generated operation signal to the control section 910.

The bus 912 interconnects the tuner 902, the demultiplexer 903, thedecoder 904, the video signal processing section 905, the audio signalprocessing section 907, the external interface 909, and the controlsection 910.

In the television 900 configured in this manner, the decoder 904 has afunction of the image decoding device 60 according to the embodimentdescribed above. Accordingly, also in the case of the image decoding inthe television 900, it is possible to enhance the parallelism ofdeblocking filter processes and ensure high-speed processing.

7-2. Second Example Application

FIG. 47 is a block diagram showing an example of a schematicconfiguration of a mobile phone adopting the embodiment described above.A mobile phone 920 includes an antenna 921, a communication section 922,an audio codec 923, a speaker 924, a microphone 925, a camera section926, an image processing section 927, a demultiplexing section 928, arecording/reproduction section 929, a display section 930, a controlsection 931, an operation section 932, and a bus 933.

The antenna 921 is connected to the communication section 922. Thespeaker 924 and the microphone 925 are connected to the audio codec 923.The operation section 932 is connected to the control section 931. Thebus 933 interconnects the communication section 922, the audio codec923, the camera section 926, the image processing section 927, thedemultiplexing section 928, the recording/reproduction section 929, thedisplay section 930, and the control section 931.

The mobile phone 920 performs operation such as transmission/receptionof audio signal, transmission/reception of emails or image data, imagecapturing, recording of data, and the like, in various operation modesincluding an audio communication mode, a data communication mode, animage capturing mode, and a videophone mode.

In the audio communication mode, an analogue audio signal generated bythe microphone 925 is supplied to the audio codec 923. The audio codec923 converts the analogue audio signal into audio data, and A/D convertsand compresses the converted audio data. Then, the audio codec 923outputs the compressed audio data to the communication section 922. Thecommunication section 922 encodes and modulates the audio data, andgenerates a transmission signal. Then, the communication section 922transmits the generated transmission signal to a base station (notshown) via the antenna 921. Also, the communication section 922amplifies a wireless signal received via the antenna 921 and convertsthe frequency of the wireless signal, and acquires a received signal.Then, the communication section 922 demodulates and decodes the receivedsignal and generates audio data, and outputs the generated audio data tothe audio codec 923. The audio codec 923 extends and D/A converts theaudio data, and generates an analogue audio signal. Then, the audiocodec 923 supplies the generated audio signal to the speaker 924 andcauses the audio to be output.

Also, in the data communication mode, the control section 931 generatestext data that makes up an email, according to an operation of a uservia the operation section 932, for example. Moreover, the controlsection 931 causes the text to be displayed on the display section 930.Furthermore, the control section 931 generates email data according to atransmission instruction of the user via the operation section 932, andoutputs the generated email data to the communication section 922. Then,the communication section 922 encodes and modulates the email data, andgenerates a transmission signal. Then, the communication section 922transmits the generated transmission signal to a base station (notshown) via the antenna 921. Also, the communication section 922amplifies a wireless signal received via the antenna 921 and convertsthe frequency of the wireless signal, and acquires a received signal.Then, the communication section 922 demodulates and decodes the receivedsignal, restores the email data, and outputs the restored email data tothe control section 931. The control section 931 causes the displaysection 930 to display the contents of the email, and also, causes theemail data to be stored in the storage medium of therecording/reproduction section 929.

The recording/reproduction section 929 includes an arbitrary readableand writable storage medium. For example, the storage medium may be abuilt-in storage medium such as an RAM, a flash memory or the like, oran externally mounted storage medium such as a hard disk, a magneticdisk, a magneto-optical disk, an optical disc, an USB memory, a memorycard, or the like.

Furthermore, in the image capturing mode, the camera section 926captures an image of a subject, generates image data, and outputs thegenerated image data to the image processing section 927, for example.The image processing section 927 encodes the image data input from thecamera section 926, and causes the encoded stream to be stored in thestorage medium of the recording/reproduction section 929.

Furthermore, in the videophone mode, the demultiplexing section 928multiplexes a video stream encoded by the image processing section 927and an audio stream input from the audio codec 923, and outputs themultiplexed stream to the communication section 922, for example. Thecommunication section 922 encodes and modulates the stream, andgenerates a transmission signal. Then, the communication section 922transmits the generated transmission signal to a base station (notshown) via the antenna 921. Also, the communication section 922amplifies a wireless signal received via the antenna 921 and convertsthe frequency of the wireless signal, and acquires a received signal.These transmission signal and received signal may include an encoded bitstream. Then, the communication section 922 demodulates and decodes thereceived signal, restores the stream, and outputs the restored stream tothe demultiplexing section 928. The demultiplexing section 928 separatesa video stream and an audio stream from the input stream, and outputsthe video stream to the image processing section 927 and the audiostream to the audio codec 923. The image processing section 927 decodesthe video stream, and generates video data. The video data is suppliedto the display section 930, and a series of images is displayed by thedisplay section 930. The audio codec 923 extends and D/A converts theaudio stream, and generates an analogue audio signal. Then, the audiocodec 923 supplies the generated audio signal to the speaker 924 andcauses the audio to be output.

In the mobile phone 920 configured in this manner, the image processingsection 927 has a function of the image encoding device 10 and the imagedecoding device 60 according to the embodiment described above.Accordingly, also in the case of the image decoding and encoding in themobile phone 920, it is possible to enhance the parallelism ofdeblocking filter processes and ensure high-speed processing.

7-3. Third Example Application

FIG. 48 is a block diagram showing an example of a schematicconfiguration of a recording/reproduction device adopting the embodimentdescribed above. A recording/reproduction device 940 encodes, andrecords in a recording medium, audio data and video data of a receivedbroadcast program, for example. The recording/reproduction device 940may also encode, and record in the recording medium, audio data andvideo data acquired from another device, for example. Furthermore, therecording/reproduction device 940 reproduces, using a monitor or aspeaker, data recorded in the recording medium, according to aninstruction of a user, for example. At this time, therecording/reproduction device 940 decodes the audio data and the videodata.

The recording/reproduction device 940 includes a tuner 941, an externalinterface 942, an encoder 943, an HDD (Hard Disk Drive) 944, a discdrive 945, a selector 946, a decoder 947, an OSD (On-Screen Display)948, a control section 949, and a user interface 950.

The tuner 941 extracts a signal of a desired channel from broadcastsignals received via an antenna (not shown), and demodulates theextracted signal. Then, the tuner 941 outputs an encoded bit streamobtained by demodulation to the selector 946. That is, the tuner 941serves as transmission means of the recording/reproduction device 940.

The external interface 942 is an interface for connecting therecording/reproduction device 940 and an external appliance or anetwork. For example, the external interface 942 may be an IEEE 1394interface, a network interface, an USB interface, a flash memoryinterface, or the like. For example, video data and audio data receivedby the external interface 942 are input to the encoder 943. That is, theexternal interface 942 serves as transmission means of therecording/reproduction device 940.

In the case the video data and the audio data input from the externalinterface 942 are not encoded, the encoder 943 encodes the video dataand the audio data. Then, the encoder 943 outputs the encoded bit streamto the selector 946.

The HDD 944 records in an internal hard disk an encoded bit stream,which is compressed content data of a video or audio, various programs,and other pieces of data. Also, the HDD 944 reads these pieces of datafrom the hard disk at the time of reproducing a video or audio.

The disc drive 945 records or reads data in a recording medium that ismounted. A recording medium that is mounted on the disc drive 945 may bea DVD disc (a DVD-Video, a DVD-RAM, a DVD-R, a DVD-RW, a DVD+, a DVD+RW,or the like), a Blu-ray (registered trademark) disc, or the like, forexample.

The selector 946 selects, at the time of recording a video or audio, anencoded bit stream input from the tuner 941 or the encoder 943, andoutputs the selected encoded bit stream to the HDD 944 or the disc drive945. Also, the selector 946 outputs, at the time of reproducing a videoor audio, an encoded bit stream input from the HDD 944 or the disc drive945 to the decoder 947.

The decoder 947 decodes the encoded bit stream, and generates video dataand audio data. Then, the decoder 947 outputs the generated video datato the OSD 948. Also, the decoder 904 outputs the generated audio datato an external speaker.

The OSD 948 reproduces the video data input from the decoder 947, anddisplays a video. Also, the OSD 948 may superimpose an image of a GUI,such as a menu, a button, a cursor or the like, for example, on adisplayed video.

The control section 949 includes a processor such as a CPU, and a memorysuch as an RAM or an ROM. The memory stores a program to be executed bythe CPU, program data, and the like. A program stored in the memory isread and executed by the CPU at the time of activation of therecording/reproduction device 940, for example. The CPU controls theoperation of the recording/reproduction device 940 according to anoperation signal input from the user interface 950, for example, byexecuting the program.

The user interface 950 is connected to the control section 949. The userinterface 950 includes a button and a switch used by a user to operatethe recording/reproduction device 940, and a receiving section for aremote control signal, for example. The user interface 950 detects anoperation of a user via these structural elements, generates anoperation signal, and outputs the generated operation signal to thecontrol section 949.

In the recording/reproduction device 940 configured in this manner, theencoder 943 has a function of the image encoding device 10 according tothe embodiment described above. Also, the decoder 947 has a function ofthe image decoding device 60 according to the embodiment describedabove. Accordingly, also in the case of the image decoding and encodingin the recording/reproduction device 940, it is possible to enhance theparallelism of deblocking filter processes and ensure high-speedprocessing.

7-4. Fourth Example Application

FIG. 49 is a block diagram showing an example of a schematicconfiguration of an image capturing device adopting the embodimentdescribed above. An image capturing device 960 captures an image of asubject, generates an image, encodes the image data, and records theimage data in a recording medium.

The image capturing device 960 includes an optical block 961, an imagecapturing section 962, a signal processing section 963, an imageprocessing section 964, a display section 965, an external interface966, a memory 967, a media drive 968, an OSD 969, a control section 970,a user interface 971, and a bus 972.

The optical block 961 is connected to the image capturing section 962.The image capturing section 962 is connected to the signal processingsection 963. The display section 965 is connected to the imageprocessing section 964. The user interface 971 is connected to thecontrol section 970. The bus 972 interconnects the image processingsection 964, the external interface 966, the memory 967, the media drive968, the OSD 969, and the control section 970.

The optical block 961 includes a focus lens, an aperture stop mechanism,and the like. The optical block 961 forms an optical image of a subjecton an image capturing surface of the image capturing section 962. Theimage capturing section 962 includes an image sensor such as a CCD, aCMOS or the like, and converts by photoelectric conversion the opticalimage formed on the image capturing surface into an image signal whichis an electrical signal. Then, the image capturing section 962 outputsthe image signal to the signal processing section 963.

The signal processing section 963 performs various camera signalprocesses, such as knee correction, gamma correction, color correctionand the like, on the image signal input from the image capturing section962. The signal processing section 963 outputs the image data after thecamera signal process to the image processing section 964.

The image processing section 964 encodes the image data input from thesignal processing section 963, and generates encoded data. Then, theimage processing section 964 outputs the generated encoded data to theexternal interface 966 or the media drive 968. Also, the imageprocessing section 964 decodes encoded data input from the externalinterface 966 or the media drive 968, and generates image data. Then,the image processing section 964 outputs the generated image data to thedisplay section 965. Also, the image processing section 964 may outputthe image data input from the signal processing section 963 to thedisplay section 965, and cause the image to be displayed. Furthermore,the image processing section 964 may superimpose data for displayacquired from the OSD 969 on an image to be output to the displaysection 965.

The OSD 969 generates an image of a GUI, such as a menu, a button, acursor or the like, for example, and outputs the generated image to theimage processing section 964.

The external interface 966 is configured as an USB input/outputterminal, for example. The external interface 966 connects the imagecapturing device 960 and a printer at the time of printing an image, forexample. Also, a drive is connected to the external interface 966 asnecessary. A removable medium, such as a magnetic disk, an optical discor the like, for example, is mounted on the drive, and a program readfrom the removable medium may be installed in the image capturing device960. Furthermore, the external interface 966 may be configured as anetwork interface to be connected to a network such as a LAN, theInternet or the like. That is, the external interface 966 serves astransmission means of the image capturing device 960.

A recording medium to be mounted on the media drive 968 may be anarbitrary readable and writable removable medium, such as a magneticdisk, a magneto-optical disk, an optical disc, a semiconductor memory orthe like, for example. Also, a recording medium may be fixedly mountedon the media drive 968, configuring a non-transportable storage sectionsuch as a built-in hard disk drive or an SSD (Solid State Drive), forexample.

The control section 970 includes a processor such as a CPU, and a memorysuch as an RAM or an ROM. The memory stores a program to be executed bythe CPU, program data, and the like. A program stored in the memory isread and executed by the CPU at the time of activation of the imagecapturing device 960, for example. The CPU controls the operation of theimage capturing device 960 according to an operation signal input fromthe user interface 971, for example, by executing the program.

The user interface 971 is connected to the control section 970. The userinterface 971 includes a button, a switch and the like used by a user tooperate the image capturing device 960, for example. The user interface971 detects an operation of a user via these structural elements,generates an operation signal, and outputs the generated operationsignal to the control section 970.

In the image capturing device 960 configured in this manner, the imageprocessing section 964 has a function of the image encoding device 10and the image decoding device 60 according to the embodiment describedabove. Accordingly, in the case of the image decoding and encoding inthe image capturing device 960, it is possible to enhance theparallelism of deblocking filter processes and ensure high-speedprocessing.

8. Summing-Up

With reference to FIGS. 1 through 49, there have been described threeworking examples of the deblocking filters for the image encoding device10 and the image decoding device 60 according to an embodiment. Thethree working examples use pixel values input to the deblocking filtersto determine whether the deblocking filters for a vertical boundary anda horizontal boundary need to be applied or not. The determinationprocesses can be performed in parallel independently of filteringprocess results. The dependency of processes between macro blocks iseliminated to enable parallelization of processes on macro blocks. As aresult, it is possible to avoid delay or data rate degradation due to alarge processing amount of the deblocking filters and ensure high-speedprocessing. The parallelism and sequences of deblocking filter processescan be flexibly configured according to various conditions such as imagesizes or installation environment.

According to the first working example, the determination of whether toapply the deblocking filter to vertical boundaries uses pixel values ofpixels belonging to rows to which the definition for horizontalboundaries is not applied. The determination of whether to apply thedeblocking filter to horizontal boundaries uses pixel values of pixelsbelonging to rows to which the definition for vertical boundaries is notapplied. The filtering process on a block does not update a pixel valueused for the filtering need determination process on another block.Pixel values input to the deblocking filter need not be stored in extramemory even if a filtering need determination process on a given blockfollows a filtering process on another block. This can save hardwarecosts needed for the apparatus installation.

According to the second working example, the memory not updated by thefiltering process stores a pixel value input to the deblocking filter.The filtering need determination processes reference the input pixelvalue. In this case, the filtering need determination processesreference a pixel whose position is free from restrictions. This enablesto use flexible determination conditions appropriate to various purposessuch as more accurate determination of filtering necessity or thedetermination with reduced processing costs.

According to the third working example, filtering processes for verticalboundaries and horizontal boundaries filter pixels input to thedeblocking filter. This configuration can parallelize filteringprocesses for vertical boundaries and horizontal boundaries each other.This can further accelerate processes of the deblocking filter. Anoutput pixel value is calculated based on two filter outputs in terms ofa pixel updated by two filtering processes performed in parallel.Parallelizing two filtering processes can also appropriately reduceblock distortion appearing on the vertical boundary and the horizontalboundary. An output pixel value can be calculated as a weighted averageof two filter outputs. This can allow the deblocking filter to moreeffectively eliminate the block distortion and further improve the imagequality.

The specification has mainly described examples where filteringprocesses for vertical boundaries precedes filtering processes forhorizontal boundaries. In addition, the above-described effects of thetechnology according to the disclosure are equally available to a casewhere filtering processes for horizontal boundaries precede filteringprocesses for vertical boundaries. The deblocking filter processing unitor the macro block may be sized otherwise than described in thespecification. An available technique may omit the filtering needdetermination processes and parallelize application of the deblockingfilter to vertical boundaries and horizontal boundaries.

A technique of transmitting information used for deblocking filterprocess parallelization from the encoding side to the decoding side isnot limited to the technique of multiplexing the information into theencoded stream header. For example, the information may not bemultiplexed into an encoded bit stream but may be transmitted orrecorded as separate data associated with the encoded bit stream. Theterm “association” signifies ensuring possibility of linking an image(or part of an image such as a slice or a block) contained in the bitstream with information corresponding to the image. Namely, theinformation may be transmitted over a transmission path different fromthat used for images (or bit streams). The information may be recordedon a recording medium (or a different recording area on the samerecording medium) different from that used for images (or bit streams).The information and the image (or bit stream) may be associated witheach other based on any units such as multiple frames, one frame, orpart of a frame.

The preferred embodiments of the present invention have been describedabove with reference to the accompanying drawings, whilst the presentinvention is not limited to the above examples, of course. A personskilled in the art may find various alternations and modificationswithin the scope of the appended claims, and it should be understoodthat they will naturally come under the technical scope of the presentinvention.

The specification represents filtering processes for vertical boundariesas “horizontal filtering” and filtering processes for horizontalboundaries as “vertical filtering.” Generally, filtering processes forvertical boundaries uses horizontally positioned filter taps. Filteringprocesses for horizontal boundaries uses vertically positioned filtertaps. For this reason, the above-described nomenclature is used for thefiltering processes.

REFERENCE SIGNS LIST

-   10, 60 image processing device-   112-1 to 112-n, 212-1 to 212-n first determination section (vertical    boundary-   determination section)-   114-1 to 114-n, 214-1 to 214-n second determination section    (horizontal boundary determination section)-   132-1 to 132-n, 332-1 to 332-n first filtering section (horizontal    filtering section)-   142-1 to 142-n, 342-1 to 342-n second filtering section (vertical    filtering section)-   150 parallelization control section-   208 line memory (memory)-   360 calculation section

The invention claimed is:
 1. An image processing device comprising:processing circuitry configured to perform horizontal filteringdetermination processes across a plurality of vertical block boundariesamong a plurality of blocks within a locally decoded image, eachhorizontal filtering determination process determines whether to apply ahorizontal filter to pixels neighboring a respective vertical blockboundary, apply, in parallel, the horizontal filter to the pixels thatneighbor the plurality of vertical block boundaries within the image towhich the horizontal filter is determined to be applied by thehorizontal filtering determination processes to generate a firstfiltered image, the horizontal filter being applied to each of theplurality of vertical boundaries within the image to which thehorizontal filter is determined to be applied by the horizontalfiltering determination processes without depending on horizontal filterresults of the other vertical block boundaries of the plurality ofblocks to which the horizontal filter is determined to be applied,perform vertical filtering determination processes across a plurality ofhorizontal block boundaries among the plurality of blocks within thefirst filtered image, each vertical filtering determination processdetermines whether to apply a vertical filter to pixels that neighbor arespective horizontal block boundary, apply, in parallel, the verticalfilter to the pixels that neighbor the horizontal block boundarieswithin the first filtered image generated by the horizontal filtering towhich the vertical filter is determined to be applied by the verticalfiltering determination processes to generate a second filtered image,the vertical filter being applied to each of the plurality of horizontalboundaries within the first filtered image to which the vertical filteris determined to be applied without depending on vertical filter resultsfor the other horizontal block boundaries of the plurality of blocks towhich the vertical filter is determined to be applied, and encode animage using the second filtered image.
 2. The image processing deviceaccording to claim 1, wherein each horizontal filtering determinationprocess determines whether to apply the horizontal filter to the pixelsthat neighbor the respective vertical block boundary without dependingon the horizontal filtering determination processes for other remainingvertical block boundaries.
 3. The image processing device according toclaim 2, wherein each vertical filtering determination processdetermines whether to apply the vertical filter to the pixels thatneighbor the respective horizontal block boundary without depending onthe vertical filtering determination processes for other remaininghorizontal block boundaries.
 4. The image processing device according toclaim 1, wherein the processing circuitry is configured to encode theimage per a unit of a hierarchized block.
 5. An image processing methodcomprising: performing with circuitry horizontal filtering determinationprocesses across a plurality of vertical block boundaries among aplurality of blocks within a locally decoded image, each horizontalfiltering determination process determines whether to apply a horizontalfilter to pixels neighboring a respective vertical block boundary;applying, in parallel, the horizontal filter to the pixels that neighborthe plurality of vertical block boundaries within the image to which thehorizontal filter is determined to be applied by the horizontalfiltering determination processes to generate a first filtered image,the horizontal filter being applied to each of the plurality of verticalboundaries within the image to which the horizontal filter is determinedto be applied by the horizontal filtering determination processeswithout depending on horizontal filter results of the other verticalblock boundaries of the plurality of blocks to which the horizontalfilter is determined to be applied; performing with the circuitryvertical filtering determination processes across a plurality ofhorizontal block boundaries among the plurality of blocks within thefirst filtered image, each vertical filtering determination processdetermines whether to apply a vertical filter to pixels that neighbor arespective horizontal block boundary; applying, in parallel, thevertical filter to the pixels that neighbor the horizontal blockboundaries within the first filtered image generated by the horizontalfiltering to which the vertical filter is determined to be applied bythe vertical filtering determination processes to generate a secondfiltered image, the vertical filter being applied to each of theplurality of horizontal boundaries within the first filtered image towhich the vertical filter is determined to be applied without dependingon vertical filter results for the other horizontal block boundaries ofthe plurality of blocks to which the vertical filter is determined to beapplied; and encoding an image using the second filtered image.
 6. Theimage processing method of claim 5, wherein each horizontal filteringdetermination process includes determining whether to apply thehorizontal filter to the pixels that neighbor the respective verticalblock boundary without depending on the horizontal filteringdetermination processes for other remaining vertical block boundaries.7. The image processing method of claim 6, wherein each verticalfiltering determination process includes determining whether to applythe vertical filter to the pixels that neighbor the respectivehorizontal block boundary without depending on the vertical filteringdetermination processes for other remaining horizontal block boundaries.8. The image processing method of claim 5, wherein the decoding includesencoding the image per a unit of a hierarchized block.